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Abstract 

Air  Force  Civil  Engineer  information  systems  are  subject  to  the  same  issues 
plaguing  civilian  information  systems.  Maximized  return  on  information  system 
investments  is  not  realized  due  to  low  technology  acceptance  by  end  users.  Contributing 
to  acceptance  is  ease  of  use,  and  one  way  to  raise  acceptance  of  information  systems  is  to 
increase  their  usability.  It  was  proposed  that  low  usability  of  these  information  systems 
resulted  from  non-  or  partial- specification  of  usability  engineering  principles  in  the 
design  of  Civil  Engineer  information  systems. 

A  case  study  methodology  was  used  in  accomplishing  this  research.  A  literature 
review  verified  that  usability  engineering  principles  were,  indeed,  non-  or  partially- 
specified  by  Air  Force  regulations  and  guidance.  An  information  system  representative 
of  other  Civil  Engineer  information  systems  was  inspected  using  the  heuristic  usability 
inspection  method.  The  results  of  this  inspection  showed  the  representative  system  to  be 
highly  usable  from  the  perspective  of  usability  engineering  principles  specified  by 
regulations  and  guidance  but  low  in  usability  in  all  other  usability  engineering  principles. 
The  results  of  the  heuristic  inspection  method  were  used  to  provide  recommendations  for 
improving  the  usability  of  Air  Force  Civil  Engineer  information  systems  in  order  to 
maximize  the  acceptance  of  these  systems  by  end  users. 
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USABLE  DESIGN  OF  CIVIL  ENGINEER  INFORMATION  SYSTEMS 


I.  Introduction 

In  today’s  technology-dependent  business  world,  information  system  design  is  no 
longer  just  a  means  toward  generating  modem  and  effective  decision  aids,  but  a  process 
of  organizational  improvement.  Information  systems  have  become  so  integrated  into 
business  operations  that  the  effective  design  of  efficient  systems  has  become  critical  to 
maintaining  a  high  tempo  of  everyday  processes.  Often  these  systems  are  designed  with 
functional  features  in  mind,  yet,  as  important  as  these  functional  features  are,  there  are 
other  factors  that  contribute  to  the  effectiveness  of  information  systems.  System 
developers  and  program  managers  must  realize  that  the  success  of  information  systems 
lies  not  simply  in  function,  but  in  the  end  users’  ability  to  easily  use  these  features.  This 
research  intends  to  qualify  the  importance  of  usability  in  information  system  design. 

Background 

A  good  information  system  is  only  effective  if  usable  by  people.  If  people  cannot 
use  the  system  to  gather  information  to  aid  in  decision-making,  then  the  information 
system  is  irrelevant.  Even  if  users  can  use  a  system,  a  much  greater  challenge  is  making 
users  want  to  use  a  system.  An  entire  field  of  research,  called  technology  acceptance,  is 
dedicated  to  this  subject.  In  addition,  another  field  of  research,  usability  engineering,  is 
dedicated  to  maximizing  one  of  the  main  factors  contributing  to  technology  acceptance: 


1 


ease  of  use.  Finally,  a  product  is  only  as  good  as  its  design,  and  designs  are  governed  by 
standards.  This  research  will  explore  the  role  of  standards  in  governing  usability  towards 
an  end  of  maximizing  technology  acceptance. 

Technology  Acceptance 

Technology  acceptance  is  the  search  “to  better  understand  why  people  accept  or 
reject  computers”  (Davis  et  al.,  1989:1).  This  field  of  research  grew  from  the  realization 
that  large  amounts  of  time  and  money  were  being  spent  on  information  systems  that 
ultimately  went  unused  by  the  intended  end  users.  Such  significant  investments  yielding 
minimal  returns  necessitated  research  into  the  reasons  why  information  systems  were 
failing  (Venkatesh  et  al.,  2003). 

“Understanding  and  creating  the  conditions  under  which  information  systems  will 
be  embraced  by  the  human  organization”  (Venkatesh  and  Davis,  2000:186)  is  one  way  of 
describing  the  goal  behind  the  study  of  technology  acceptance.  In  other  words,  the  study 
of  technology  acceptance  seeks  to  develop  a  better  understanding  of  what  it  takes  to 
encourage  potential  users  to  accept  (i.e.,  use)  information  systems.  By  understanding 
what  motivates  users,  system  developers  and  program  managers  can  create  information 
system  designs  that  cater  to  the  factors  that  encourage  or  otherwise  bolster  technology 
acceptance.  Organizations  benefit  from  this  because  their  technology  investments  do  not 
go  to  waste  as  the  result  of  non-usage. 

Of  the  various  technology  acceptance  theories  available  in  the  literature,  the 
Technology  Acceptance  Model  (TAM)  and  its  derivates  are  some  of  the  most  researched 
and  validated  models  (Venkatesh  and  Davis,  2000).  The  TAM  model  suggests  that  the 
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acceptance  of  technology  is  based  on  four  main  elements:  behavioral  intentions  toward 
the  technology,  attitudes  toward  the  technology,  perceived  usefulness  of  the  technology, 
and  perceived  ease  of  use  of  the  technology  (Davis  et  al.,  1989).  Validating  studies  vary 
in  their  assessments  of  the  influences  of  these  elements  on  technology  acceptance,  but 
one  thing  is  constant:  perceived  ease  of  use  has  been  consistently  cited  as  a  significant 
influence  on  technology  acceptance  (Davis  et  al.,  1989;  Venkatesh  and  Davis,  2000; 
Venkatesh  et  al.,  2002;  Venkatesh  et  al.,  2003). 

Before  proceeding  any  further,  it  is  important  to  realize  the  differences  and 
similarities  between  the  terms  “ease  of  use”  and  “usability,”  as  well  as  clarify  how  these 
terms  will  be  used  in  the  context  of  this  research.  Perceived  ease  of  use  has  been 
established  as  a  key  factor  in  user  acceptance  of  information  technology.  It  is  a  measure 
of  the  user’s  perception  that  using  a  technology  will  be  free  from  effort  (Davis  et  al., 
1989).  The  purveyor  of  this  definition,  Fred  Davis,  has  been  quoted  defining  usability  as 
having  “’ . .  .subsumed  two  constructs  -  usefulness  and  ease  of  use’”  (Garcia,  2005:1). 
Although  Davis’  definition  of  usability  includes  ease  of  use  and  usefulness  combined,  in 
practice  “...usability  testing  focuses  primarily  on  a  system’s  ease  of  use”  (Davis  and 
Venkatesh,  2004: 1).  As  will  be  clarified  in  the  following  chapters,  the  nature  of  this 
research  is  not  concerned  with  usefulness,  rather  the  goal  involves  examining  ease  of  use. 
Because  of  the  nature  of  this  research,  and  because  of  the  focus  of  usability  engineering 
on  ease  of  use,  it  is  justified,  in  the  context  of  this  research,  to  use  the  terms  “usability” 
and  “ease  of  use”  interchangeably.  With  this  in  mind,  the  concept  of  usability  from  the 
perspective  of  usability  engineering  experts  will  be  described. 
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Usability 

Usability,  as  defined  by  Merriam  Webster’s  Collegiate  Dictionary  (1994:1301),  is 
the  state  of  being  “convenient  and  practicable  for  use.”  In  terms  of  information  systems, 
Hoffer  (2002)  describes  three  key  characteristics  of  usability:  speed,  accuracy,  and 
satisfaction.  In  other  words,  how  quickly  and  accurately  users  can  manipulate  the 
system,  and  how  satisfied  they  are  with  the  output,  define  the  usability  of  a  system. 
Furthermore,  Nielsen  (Useit.com,  2003)  breaks  usability  down  into  five  characteristics: 
leamability,  efficiency,  memorability,  errors,  and  satisfaction. 

The  manifestations  of  these  characteristics  can  be  measured  with  various 
techniques.  One  of  these  techniques  is  the  heuristic  inspection  method,  which  will  be 
used  in  this  research  for  its  ability  to  provide  useful  results  with  a  simple,  economical, 
and  efficient  procedure  (Nielsen,  1994).  Heuristics  usability  inspections  involve  the 
assessment  of  the  system  in  question  with  principles  accepted  as  characteristics  of  highly 
usable  systems.  The  following  ten  principles  form  the  backbone  of  a  heuristics 
inspection:  visibility  of  system  status;  match  between  the  system  and  real  world;  user 
control  and  freedom;  consistency  and  standards;  error  prevention;  recognition  rather  than 
recall;  flexibility  and  efficiency  of  use;  aesthetic  and  minimalist  design;  ability  to  help 
users  recognize,  diagnose,  and  recover  from  errors;  and  system  help  and  documentation 
(Nielsen,  1994). 

An  important  thing  to  note  is  that  Nielsen  (1994)  states  that  for  intranet  systems, 
which  are  mandated  by  organizational  policy,  user  satisfaction  is  not  as  critical  since 
employees  have  no  other  choice  of  systems.  Since  users  are  forced  to  use  the  system, 
design  efforts  are  typically  directed  toward  reducing  errors  and  increasing  efficiency  and 
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memorability  instead  of  improving  customer  satisfaction  (Nielsen,  1994).  This  may  be 
true  if  user  satisfaction  is  viewed  as  a  unidirectional  determinant  of  usability.  The  TAM 
suggests  that  user  satisfaction  cannot  be  ignored  when  roles  are  reversed.  Usability  is 
viewed  as  a  factor  in  contributing  to  user  attitudes,  which  contribute  to  behavioral 
intention  to  use,  which  ultimately  leads  to  technology  use.  Thus,  the  full  spectrum  of 
usability  characteristics  should  be  addressed  in  the  design  process  (Davis  et  al.,  1989). 

Both  private  and  public  sector  organizations  struggle  with  the  balance  between 
customer  (i.e.,  external)  and  user  (i.e.,  internal)  satisfaction.  This  is  particularly  true  in 
the  Air  Force,  where  information  technology  is  often  viewed  from  a  mandatory  use 
perspective  and  thus  minimally  addresses  the  satisfaction  characteristic  associated  with 
usable  systems.  In  addition,  current  Air  Force  standards  mainly  address  the  consistency 
and  standards  principle  of  usability.  This  is  reflected  in  the  system  engineering  process 
typical  of  most  Civil  Engineer  information  systems.  To  gain  a  better  appreciation  for 
design  of  information  technology  systems  in  the  Air  Force,  it  is  useful  to  briefly  review 
the  existing  standards. 

Existing  Air  Force  Standards 

The  review  of  existing  standards  involved  searching  through  four  frameworks  of 
information  system  design  that  have  been  applicable  to  Civil  Engineer  information 
system  design  in  recent  years.  One  of  the  most  recent  frameworks,  the  Global  Combat 
Support  System,  was  found  to  focus  more  on  integration  of  various  combat  support 
information  systems  and  less  on  design  standardization.  A  superseded  framework,  the 
Technical  Architecture  Framework  for  Information  Management  (TAFIM),  was  found  to 
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contain  an  individual  volume,  the  Human-Computer  Interface  Style  Guide  (or  simply, 
“Style  Guide”),  that  was,  at  one  time,  applicable  to  standardization  of  design,  and  more 
specifically,  applicable  to  the  design  of  the  information  system  used  in  the  methodology 
of  this  research.  Finally,  two  evolutions  of  frameworks,  the  Joint  Technical  Architecture 
(JTA)  and  Department  of  Defense  (DoD)  Information  Technology  Standards  Registry 
(DISR),  were  found  to  be  readily  applicable  as  standards  for  current  information  system 
design  efforts.  Of  these  frameworks,  the  latter  three  were  found  to  be  relevant  to  this 
research  and  were  scoured  to  make  assessments  about  the  extent  of  usability  engineering 
principles  that  are  specified  in  the  design  of  Civil  Engineer  information  systems. 

The  GCSS  was  developed  with  the  overall  purpose  of  integrating  stove-piped 
DoD  information  systems  such  that  they  can  be  accessed  by  any  authorized  user  in  any 
location  using  commonly  available  equipment  (DISA,  2005).  With  such  a  purpose  in 
mind,  it  came  as  no  surprise  that  the  GCSS  displayed  a  focus  on  integration  and  not  on 
design  regulations  and  guidance  geared  toward  standardization.  Therefore,  the  researcher 
decided  that  the  GCSS  framework  was  not  applicable  in  the  context  of  this  research, 
since  its  design  focus  was  centered  on  the  interfaces  between  information  systems  rather 
than  the  interfaces  between  information  systems  and  end  users. 

A  review  of  the  TAFIM  Style  Guide  showed  that  this  document  specified 
primarily  human-computer  interface  features.  The  recommendations  in  the  TAFIM  Style 
Guide  are  mainly  directed  toward  ensuring  consistency  of  design  features  across  all 
information  system  elements.  Some  examples  of  information  system  elements  addressed 
include  keyboard  layout,  screen  design,  and  menu  appearance.  The  main  point  to  realize 
here  is  that  the  TAFIM  Style  Guide  was  intended  to  maximize  ease  of  use  by  minimizing 
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the  diversity  of  human-computer  interface  methods. 

A  review  of  the  JTA  and  DISR  revealed  a  lack  of  standards  for  usability 
engineering  in  DoD  information  systems.  Explicit  human-computer  interface  standards, 
similar  to  those  specified  in  the  TAFIM  Style  Guide,  are  listed  for  weapon  system  HCI 
and  nuclear  system  HCI;  however,  no  standards  were  found  for  information  systems  in 
general  (Disronline.disa.mil,  2004). 

The  review  of  standards  is  further  qualified  in  Chapter  2  of  this  research.  Based 
on  review  of  the  four  frameworks  mentioned  above,  it  was  proposed  that  a  gap  could  be 
visualized  between  current  standards  and  an  optimal  usability  standard.  In  the  context  of 
this  research,  an  optimal  usability  standard  would  be  written  in  such  a  manner  that  each 
of  the  characteristics  of  highly  usable  systems  (learnability,  efficiency,  memorability,  free 
from  errors,  and  satisfaction)  are  maximized  by  addressing  the  ten  usability  principles 
(visibility  of  system  status;  match  between  the  system  and  real  world;  user  control  and 
freedom;  consistency  and  standards;  error  prevention;  recognition  rather  than  recall; 
flexibility  and  efficiency  of  use;  aesthetic  and  minimalist  design;  ability  to  help  users 
recognize,  diagnose,  and  recover  from  errors;  and  system  help  and  documentation) 
(Nielsen,  1994).  The  researcher  found  that  applicable  design  standards  addressing 
usability  engineering  are  either  non-existent  or  those  that  do  exist  address  mainly  the 
consistency  and  standards  principle  of  usability  engineering.  It  is  this  observation  that 
led  to  the  proposition  that  a  gap  exists  between  current  standards  and  an  ideal  standard. 
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Problem  Statement 


Usability  is  a  key  requirement  of  information  systems  (Nielsen,  1994  and  2000). 
It  leads  to  positive  user  attitudes,  which  ultimately  lead  to  acceptance  and  use  of 
information  systems  (Davis  et  al.,  1989).  Standards  applicable  to  the  design  of  Civil 
Engineer  information  systems  focus  on  only  one  of  the  ten  usability  engineering 
principles,  consistency  and  standards.  Based  on  the  fact  that  nine  usability  engineering 
principles  are  not  required  by  Air  Force  regulation  and  guidance,  the  researcher  proposes 
that  usability  of  Civil  Engineer  information  systems  is  not  optimal.  The  propositions  of 
this  research  are  formally  stated  below: 

1.  Usability  in  Civil  Engineer  information  systems  is  not  optimal. 

2.  Non-optimal  usability  in  Civil  Engineer  information  systems  can  be  linked  to 
non- specification  of  usability  engineering  principles  in  Civil  Engineer 
information  system  regulations  and  guidance. 

While  propositions  serve  to  outline  the  background  of  a  given  study,  research  questions 

provide  the  level  of  detail  necessary  to  actually  implement  the  research  methodology  in 

order  to  validate  the  propositions. 

Research  Questions 

To  address  the  propositions  serving  as  the  problem  statement  of  this  research, 

questions  must  be  posed  that  can  be  answered  by  the  execution  of  a  methodology.  As 

related  to  this  research,  the  questions  are: 

1 .  How  do  current  standards  related  to  the  design  of  Civil  Engineer  information 
systems  specify  usability  engineering  principles? 
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2.  How  are  gaps  in  existing  Civil  Engineer  information  system  design  standards 
qualified  upon  observation  of  Civil  Engineer  information  systems  through 
proven  and  accepted  usability  inspection  methods? 

3.  How  can  improvements  to  usability  be  made  as  a  result  of  any  findings 
yielded  by  the  usability  inspection  of  a  Civil  Engineer  information  system? 

The  research  questions  clarify  the  goals  that  will  be  achieved  through  execution  of  the 

methodology.  On  overview  of  the  methodology  of  this  research  is  provided  in  the  next 

section. 

Methodology 

The  researcher  considered  various  research  methods  as  possible  ways  to  answer 
the  research  questions.  The  methods  considered  were  naturalistic  observation,  the  case 
study  method  of  observation,  correlational  research,  differential  methods,  and 
experimental  methods.  For  reasons  detailed  in  Chapter  3  of  this  research,  the  case  study 
research  method  was  chosen. 

As  part  of  the  case  study  methodology,  a  literature  review  was  conducted  to 
answer  the  first  research  question.  In  order  to  answer  the  remaining  research  questions, 
the  methodology  had  to  be  further  characterized  by  choosing  a  case  study  type.  A 
holistic,  single-case  study  type  was  chosen  for  reasons  outlined  in  Chapter  3.  In  this  type 
of  case  study,  there  is  a  single  unit  of  analysis  that  does  not  have  any  subtypes  or 
embedded  units.  The  unit  of  analysis  is  then  observed  in  order  to  witness  any  behaviors 
that  might  answer  the  research  questions.  The  unit  of  analysis  had  to  be  carefully 
selected  such  that  it  would  provide  the  opportunity  to  observe  any  effects  of  the  non- 
optimal  usability  standards  stated  in  the  research  propositions.  The  unit  of  analysis 
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chosen  was  the  Automated  Civil  Engineer  System  Personnel  Readiness  (ACES -PR) 
module,  since  it  is  an  information  system  representative  of  other  Civil  Engineer 
information  systems  and  has  shown  evidence  of  usability  problems  in  the  past.  Equally 
as  relevant  as  the  unit  of  analysis  is  the  method  of  observation.  The  case  study 
instrument  used  in  this  research  was  the  heuristic  usability  inspection  method  (Nielsen, 
1994)  as  embodied  by  the  checklist  created  by  Pierotti  (2002).  Using  the  heuristic 
inspection  method  on  a  suitable  unit  of  analysis  allowed  for  the  second  and  third  research 
questions  to  be  answered. 

Assumptions  and  Limitations 

One  very  significant  assumption  made  in  this  research  is  that  Civil  Engineer 
information  systems  are  subject  to  a  global  set  of  design  standards,  and  that  the  design 
process  of  Civil  Engineer  information  systems  adhered  to  these  standards.  This 
assumption  is  important  since  one  Civil  Engineer  information  system  was  chosen  as 
representative  of  other  Civil  Engineer  information  systems  in  order  to  execute  the 
methodology  and  draw  conclusions  about  Civil  Engineer  information  system  regulations 
and  guidance  documents. 

Another  significant  limitation  of  this  research  is  that  the  number  of  usability 
inspection  methods  available  for  this  study  is  limited  by  the  manpower  available  to 
perform  the  inspections.  Some  of  the  available  methods  require  teams  of  several  people 
to  accomplish  the  required  interviews,  evaluations,  and  review  of  thousands  of  lines  of 
programming  code.  Various  time,  manpower,  funding,  and  equipment  constraints  make 
heuristic  evaluation  the  favorable  method  of  assessing  usability. 
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The  technology  acceptance  and  usability  engineering  fields  are  constantly  and 


rapidly  evolving.  Although  recent  research  (Venkatesh  et  al.,  2003)  claims  to  be  near  the 
full  capacity  of  understanding  technology  acceptance,  much  work  is  left  to  be  done.  As  a 
result,  this  research  will  provide  a  point-in-time  analysis,  and  future  research  will  be 
necessary  to  periodically  revalidate  the  roles  of  technology  acceptance  and  usability 
engineering  in  the  ACES-PR  system  engineering  process. 

Organization/Purpose  of  Remaining  Chapters 

Having  generally  outlined  the  purpose,  background,  and  methodology  of  this 
research,  a  brief  synopsis  should  be  given  of  what  remains  to  be  discussed  in  the 
following  chapters.  The  second  chapter  presents  a  more  detailed  review  of  the  research 
scenario  and  the  relevant  literature.  The  specifics  of  TAM,  usability  engineering,  and  the 
link  between  the  two;  current  DoD  standards;  the  ACES-PR  system  engineering  process; 
and  the  background  documentation  supporting  the  methodology  used  in  this  research  are 
discussed.  Chapter  3  will  specify  the  details  of  the  methodology  for  accomplishing  this 
research.  More  specifically,  heuristic  inspection  methods  will  be  discussed  in  detail.  The 
fourth  chapter  is  intended  to  summarize  the  results  of  executing  the  methodology,  with  a 
focus  on  interpretation  of  the  findings.  Finally,  Chapter  5  provides  the  research’s 
conclusions  and  identifies  areas  for  future  research. 
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II.  Literature  Review 


This  chapter  provides  an  overview  of  the  literature  pertinent  to  this  research.  The 
field  of  technology  acceptance  will  be  explored  by  reviewing  the  most  prevalent  and 
widely-recognized  technology  acceptance  models.  This  will  lead  to  an  in-depth 
discussion  of  usability  engineering.  After  reviewing  the  literature,  the  current  Air  Force 
standards  related  to  usability  will  be  discussed;  this  will  include  how  usability  can  be 
measured  and  inspected.  Since  the  Automated  Civil  Engineer  System  Personnel 
Readiness  (ACES-PR)  module  will  be  used  as  the  unit  of  analysis  for  the  methodology  of 
this  research,  a  brief  background  of  the  system  will  be  provided. 

Behavioral  Models 

Organizations  have  recognized  the  importance  of  information  technology  (IT)  and 
have  dramatically  increased  IT  investments  (Venkatesh  et  al.,  2003).  However, 
performance  gains  resulting  from  such  investments  have  been  low,  which  Davis  et  al. 
(1989)  attribute  to  users’  non-acceptance  of  IT  systems.  To  develop  a  better 
understanding  of  this  phenomenon,  researchers  have  developed  numerous  scientific 
models.  Of  the  models  discussed  below,  the  Technology  Acceptance  Model  is  the  most 
appropriate  one  for  this  research  and  will  be  explored  in  more  detail. 

Theory  of  Reasoned  Action  (TRA) 

The  basic  tenets  of  the  TRA  are  a  person’s  attitude  toward  a  target  behavior  and 
the  concept  of  subjective  norm.  The  TRA  maintains  that  a  person’s  positive  or  negative 
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feelings  about  a  behavior  determine  their  likelihood  to  perform  the  behavior.  In  addition, 
this  likelihood  is  affected  by  their  perception  of  the  subjective  norm  -  whether  or  not 
people  important  to  the  person  think  the  person  should  perform  the  behavior  (Venkatesh 
et  al.,  2003). 

Motivational  Model 

The  motivational  model  is  based  on  the  concept  of  what  motivates  people  to 
perform  a  given  activity.  It  qualifies  motivation  in  two  forms,  extrinsic  and  intrinsic. 
Extrinsic  motivation  is  the  idea  that  people  are  enticed  to  perform  a  particular  activity  by 
something  other  than  the  activity  itself.  Intrinsic  motivation  is  the  idea  that  people  are 
motivated  by  the  activity  itself.  An  example  of  an  extrinsically  motivated  behavior  is 
increased  performance  at  work  as  the  result  of  a  pay  raise,  whereas  an  intrinsically 
motivated  worker  needs  no  pay  raise  because  his  motivation  derives  from  the  work  itself 
(Venkatesh  et  al.,  2003). 

Technology  Acceptance  Model  (TAM)  Overview 

The  TAM  asserts  that  technology  usage  is  determined  by  two  major  factors: 
perceived  usefulness  and  perceived  ease  of  use.  Perceived  usefulness  describes  a 
person’s  perception  that  a  given  technology  will  increase  the  person’s  job  performance. 
Perceived  ease  of  use  describes  a  person’s  perception  that  a  technology  will  be  free  from 
effort.  Both  of  these  factors  have  been  widely  validated  as  having  a  positive  correlation 
with  technology  usage  (Venkatesh  et  al.,  2003). 
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Theory  of  Planned  Behavior  ( TPB ) 

The  TPB  is  an  extension  of  the  TRA.  It  adapts  the  factors  of  TRA  but  includes 
one  more,  perceived  behavioral  control.  The  TPB  holds  that,  in  addition  to  a  person’s 
attitude  and  perception  of  the  subjective  norm,  the  person’s  perception  of  the  ease  or 
difficulty  of  an  activity  also  determines  their  likelihood  to  perform  the  activity 
(Venkatesh  et  al.,  2003). 

Combined  TAM  and  TPB 

The  Combined  TAM  and  TPB  model  is  exactly  that,  a  combination  of  the  factors 
from  the  TAM  and  TPB  to  model  a  person’s  behavior  toward  technology.  Thus,  attitude, 
subjective  norm,  perceived  usefulness,  and  perceived  ease  of  use  are  all  determinants  of 
behavior  toward  technology  in  this  model  (Venkatesh  et  al.,  2003). 

Model  of  PC  Utilization 

This  model  predicts  that  six  factors  contribute  toward  a  person’s  usage  of 
technology:  job-fit,  complexity,  long-term  consequences,  affect  towards  use,  social 
factors,  and  facilitating  conditions.  Job-fit  describes  a  person’s  perception  that  a  given 
technology  will  enhance  job  performance.  Complexity  is  the  technology’s  perceived 
level  of  difficulty  in  being  understood.  Long  term  consequences  describes  the  perception 
that  a  technology  will  pay  off  in  the  long  term.  Affect  toward  use  is  the  concept  of 
feeling  human  emotion  toward  a  technology.  Social  factors  are  those  cultural  and 
interpersonal  internalizations  that  the  person  has  made  in  regard  to  the  technology. 
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Facilitating  conditions  are  those  objective  environmental  factors  that  contribute  to  a 
system’s  actual,  and  not  simply  perceived,  ease  of  use  (Venkatesh  et  al.,  2003). 

Innovation  Diffusion  Theory 

Innovation  diffusion  theory,  geared  toward  acceptance  of  new  technology, 
involves  seven  factors:  relative  advantage,  ease  of  use,  image,  visibility,  compatibility, 
results  demonstrability,  and  voluntariness  of  use.  Relative  advantage  is  achieved  if  a 
person  perceives  a  new  technology  as  better  than  its  predecessor.  Ease  of  use  describes 
the  perception  that  a  technology’s  use  will  be  free  from  effort.  The  image  factor  is  the 
perception  that  a  technology  will  increase  a  person’s  status  in  a  social  system.  A 
technology  has  visibility  if  people  can  visually  perceive  other  people  in  their  organization 
using  the  technology.  Compatibility  is  the  perception  that  a  given  technology  is 
consistent  with  a  person’s  existing  values,  needs,  and  past  experiences.  Results 
demonstrability  is  the  degree  to  which  the  person  can  tangibly  observe  the  outcomes  of 
the  technology.  Voluntariness  of  use  is  the  concept  describing  a  person’s  perception  that 
a  technology’s  use  is  on  terms  of  the  person’s  own  free  will  (Venkatesh  et  al.,  2003). 

Social  Cognitive  Theory  (SCT) 

The  SCT  is  a  theory  of  human  behavior  based  on  performance  outcome 
expectations,  personal  outcome  expectations,  self  efficacy,  affect,  and  anxiety.  In  regards 
to  technology,  performance  outcome  expectations  are  those  expectations  people  have 
about  an  activity  that  is  related  to  their  performance  at  a  job.  Personal  outcome 
expectations  are  those  expectations  people  have  about  an  activity  that  is  related  to  their 
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individual  self  esteem  or  sense  of  accomplishment.  In  the  context  of  technology,  self 
efficacy  is  the  perception  a  person  has  about  his  or  her  own  ability  to  use  the  technology 
to  accomplish  their  job.  Affect  describe’s  a  person’s  like  or  dislike  of  a  given  activity. 
Anxiety  is  a  concept  describing  the  degree  to  which  a  person  experiences  emotional 
reactions  toward  an  activity  (Venkatesh  et  al.,  2003). 

Technology  Acceptance  Model  Detailed  Perspective 

Although  all  of  the  models  seek  to  explain  important  factors  related  to  user 
acceptance  of  information  technology,  the  TAM  was  found  to  be  most  relevant  to  this 
research  since  it  has  been  widely  validated  in  terms  of  information  technology.  Many  of 
the  other  models  were  derived  in  response  to  needs  in  the  fields  of  sociology  or 
psychology,  and  validated  in  those  contexts.  The  rest  of  this  section  is  dedicated  to 
further  explaining  the  concepts  that  form  the  TAM. 

The  TAM  maintains  that  performance  gains  for  the  organization  will  not  be 
realized  if  employees  do  not  make  use  of  the  purchased  IT.  However,  for  the  IT  systems 
to  be  used,  the  systems  must  first  be  accepted  by  the  users  on  a  behavioral  level  (Davis  et 
al.,  1989).  Given  an  ideal  situation  where  a  system  is  in  the  early  stages  of  the  design 
process,  discovering  and  understanding  the  factors  that  contribute  to  user  acceptance  can 
help  system  developers  create  more  effective  IT  systems.  Alternately,  if  a  system  is 
already  deployed,  understanding  user  acceptance  factors  can  lead  to  better  redesign 
efforts  in  future  versions  of  the  system.  Two  major  user  acceptance  factors,  perceived 
usefulness  and  perceived  ease  of  use,  were  identified  in  the  beginning  of  the  TAM 
research  (Davis  et  al.,  1989)  and  have  consistently  been  included  in  many  prominent 
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studies  validating  the  TAM  (e.g.,  Adams  et  al.,  1992;  Hendrickson  et  al.,  1993;  Szajna, 
1994;  Agarwal  and  Prasad,  1999;  Malhotra  and  Galetta,  1999;  Venkatesh  and  Davis, 
2000;  Venkatesh  et  al.,  2002;  Venkatesh  et  al.,  2003;  McFarland  and  Hamilton,  2004;  and 
Venkatesh  and  Davis,  2004).  In  these  studies,  perceived  usefulness  and  perceived  ease  of 
use  were  found  to  have  a  significant  positive  correlation  with  technology  acceptance  (i.e., 
actual  system  use).  Additionally,  most  of  the  listed  studies  accept  the  core  factor 
relationships  outlined  by  Davis  et  al.  (1989)  during  the  development  of  the  TAM. 

As  shown  in  Figure  1,  the  original  TAM  model  defines  five  core  factors  and  their 
relationships  contributing  to  technology  acceptance.  Perceived  usefulness  is  a  subjective 
factor  describing  the  perception  of  a  user  that  a  particular  IT  system  will  increase  job 
performance  as  a  result  of  its  use.  Perceived  ease  of  use  is  also  a  subjective  factor;  it 
describes  the  user’s  perception  that  using  a  particular  IT  system  will  be  free  from  effort. 
Theses  two  factors  are  influenced  by  external  variables,  which  include  such  things  as 
“system  design  characteristics,  user  characteristics,  nature  of  the  development  or 
implementation  process,  political  influences,  [and]  organizational  structure”  (Davis  et  al., 
1989:984).  The  interactions  of  these  factors  impact  the  user  attitudes  factor,  which 
describes  the  positive  or  negative  feelings  a  user  has  toward  the  technology.  Finally, 
behavioral  intention  describes  how  strong  a  user’s  intentions  are  to  actually  use  the 
system. 
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Figure  1.  Technology  Acceptance  Model  Relationships  (Davis  et  al.,  1989) 


Included  in  the  Davis  et  al.  (1989)  study  was  a  validation  of  the  interaction 
between  the  different  factors  in  the  TAM.  Business  administration  master’s  degree 
students  were  surveyed  on  their  usage  of  a  word  processing  software  package.  The 
results  verified  that  the  proposed  relationships  shown  in  Figure  1  all  had  positive 
correlations.  The  two  most  significant  factors  affecting  technology  acceptance  were 
found  to  be  perceived  usefulness  and  perceived  ease  of  use.  While  usefulness  was  found 
to  be  significant,  the  intent  of  this  research  was  not  to  suggest  more  features  or  functions 
that  will  make  IT  systems  more  useful.  Therefore,  usefulness  issues  are  not  addressed. 
Instead,  this  research  focused  on  the  ease  of  use  factor.  The  corresponding  field  of  study 
that  seeks  to  maximize  IT  system  ease  of  use  is  called  usability  engineering. 

Usability  Engineering 

Usability  can  be  defined  in  various  ways.  International  Standards  Organization 
(ISO)  Standard  9241,  Part  1 1  (1998:2),  defines  it  as  the  “extent  to  which  a  product  can  be 
used  by  specified  users  to  achieve  specified  goals  with  effectiveness,  efficiency  and 
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satisfaction  in  a  specified  context  of  use.”  Effectiveness  is  subsequently  defined  as  the 
“accuracy  and  completeness  with  which  users  achieve  specified  goals”  (ISO,  1998:2). 
Similarly,  efficiency  is  defined  as  the  “resources  expended  in  relation  to  the  accuracy  and 
completeness  with  which  users  achieve  goals”  (ISO,  1998:2).  Context  of  use  is  defined 
as  the  “users,  tasks,  equipment  (hardware,  software  and  materials),  and  the  physical  and 
social  environments  in  which  a  product  is  used”  (ISO,  1998:2).  Finally,  satisfaction  is 
“the  extent  to  which  the  user  finds  the  use  of  the  product  acceptable”  (ISO,  1998:2). 
Nielsen  (Useit.com,  2003),  often  referred  to  as  “the  reigning  guru  of  web  usability”  and 
“the  world’s  leading  expert  on  user-friendly  design,”  defines  usability  in  terms  of  five 
attributes:  leamability,  efficiency,  memorability,  errors,  and  satisfaction. 

Learnability 

The  learnability  attribute  describes  how  quickly,  given  no  previous  exposure  to  it, 
users  become  proficient  with  an  IT  system.  Beginner  users  can  become  expert  users 
quickly  in  a  system  with  high  learnability  (Useit.com,  2003).  An  example  of  a  system 
with  relatively  high  leamability  is  Microsoft  Windows  XP.  The  first  time  an  individual 
uses  a  Windows  XP  computer,  an  easily  understandable  tour  program  is  presented  to 
introduce  the  new  user  to  the  system’s  features.  When  the  tour  is  complete,  a  “tooltips” 
box  appears  whenever  the  computer  mouse  is  dragged  over  icons  appearing  on  the 
screen.  This  “tooltips”  box  tells  the  user  the  function  of  each  icon.  If  a  user  has  further 
questions,  a  product  support  feature  called  “Windows  Help”  allows  plain  text  questions 
to  be  entered  by  the  user;  relatively  clear  and  descriptive  solutions  are  then  provided  by 
the  system.  Features  such  as  the  tour,  tooltips,  and  a  help  database  are  not  sufficient  by 
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themselves  to  consider  Windows  XP  leamable;  however,  they  are  significant  initial 
contributors  to  the  leamability  of  the  system. 

Efficiency 

An  efficient  IT  system  provides  relevant  output  without  requiring  excessive  input. 
Once  a  user  has  learned  the  system,  an  efficient  system  should  operate  in  a  manner 
requiring  minimal  inputs  for  desired  outputs  (Useit.com,  2003).  One  example  of  an 
efficient  IT  system  can  be  found  in  the  internet  gaming  community,  Team  Warfare 
League  (Teamwarfare.com,  2004).  Almost  any  feature  the  website  has  to  offer  can  be 
reached  within  two  to  three  clicks  on  hyperlinks.  In  this  way,  system  users  receive 
desired  outputs  with  a  minimal  amount  of  inputs. 

Memorability 

The  memorability  attribute  relates  to  how  readily  an  infrequent  user  can  return  to 
the  IT  system  after  an  extended  period  of  non-use  and  remember  how  to  use  it.  Systems 
with  high  memorability  do  not  require  extended  re-learning  periods  for  infrequent  users 
(Useit.com,  2003).  One  way  to  increase  memorability  is  to  adopt  standards  in  system 
design.  For  example,  the  icons  used  in  the  Microsoft  Internet  Explorer  web  browser 
(Figure  2)  are  similar  to  the  icons  used  in  the  Linux  operating  system’s  Galeon  web 
browser  (Figure  3).  Since  these  icons  adhere  to  the  industry  standard,  they  are 
standardized,  simple,  and  intuitive.  This  allows  users  to  easily  re-learn  one  operating 
system’s  browser  even  though  another  operating  system  might  be  used  more  often.  In 
other  words,  an  individual  may  use  Microsoft  Windows  on  a  daily  basis  and  the  Linux 


20 


operating  system  infrequently.  When  returning  to  the  Linux  Galeon  web  browser  after  an 
extended  period  of  non-use,  the  individual  is  able  to  quickly  re-learn  how  to  use  the 
Galeon  browser. 
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Figure  2.  Microsoft  Internet  Explorer  Web  Browser 


Figure  3.  Linux  Galeon  Web  Browser 


Errors 

An  IT  system  with  low  errors  is  not  prone  to  a  high  frequency  or  severity  of 
errors.  A  low-error  system  is  relatively  free  of  bugs  and  is  structured  so  that  it  does  not 
lead  a  user  toward  committing  an  error  (Useit.com,  2003).  Microsoft  Windows  XP  is 
much  less  prone  to  committing  errors,  and  recovers  from  errors  much  better,  than  its 
previous  versions.  In  these  earlier  versions,  errors  were  frequent  and  often  resulted  in  the 
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“blue  screen  of  death.”  When  this  occurred,  users  were  provided  a  rather  cryptic  message 
and  two  options,  as  shown  in  Figure  4.  The  attempt  to  continue  usually  failed  and  users 
were  forced  to  restart  their  computers,  thus  losing  unsaved  work.  Windows  XP  usually 
alerts  users  to  the  error  that  occurred  and  almost  always  allows  them  to  continue,  as 
shown  in  Figure  5.  For  this  reason,  earlier  versions  of  Windows  were  considered  less 
usable  from  an  errors  standpoint. 

Satisfaction 

User  satisfaction  describes  how  pleased  users  are  with  the  operation  of  the  IT 
system  (Useit.com,  2003).  As  outlined  in  the  TAM,  positive  user  attitudes  (i.e., 
satisfaction)  contribute  to  technology  acceptance  (Davis  et  al.,  1989).  Users  with 
negative  attitudes  toward  an  IT  system  will  most  likely  not  accept  the  system  and  may 
not  use  it.  An  example  of  this  can  be  found  in  search  engines.  A  study  published  by  “PC 
Magazine”  (Searchengineguide.com,  2004)  showed  that  Google  had  the  highest  customer 
satisfaction,  followed  by  the  Yahoo  and  MSN  search  engines.  This  satisfaction  is 
probably  responsible  for,  or  at  least  related  to,  usage  statistics  which  show  that  Google 
leads  in  searches  per  day,  again  followed  by  Yahoo  and  MSN  (lcog.com,  2004). 
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Windows 


An  exception  06  has  occured  at  0028:C11B3ADC  in  VKD  DiskTSD(03)  + 
00001660.  This  was  called  from  0028:C11B40C8  in  VkD  voltrack(04)  + 
00000000.  It  may  be  possible  to  continue  normally. 

*  Press  any  key  to  attempt  to  continue. 

*  Press  CTRL-tflLT+RESET  to  restart  your  computer.  You  will 
lose  any  unsaved  information  in  all  applications. 

Press  any  key  to  continue 


Figure  4.  Early  Microsoft  Windows  Error  Screen 


Run  a  DLL  as  an  App 


Run  a  DLL  as  an  App  has  encountered  a  problem  and 
needs  to  close.  We  are  sorry  for  the  inconvenience. 


If  you  were  in  the  middle  of  something,  the  information  you  were  working  on 
might  be  lost. 

Please  tell  Microsoft  about  this  problem. 

We  have  created  an  error  report  that  you  can  send  to  help  us  improve 
Run  a  DLL  as  an  App.  We  will  treat  this  report  as  confidential  and 
anonymous. 

T o  see  what  data  this  error  report  contains,  click  here. 

S end  E rror  R eport  D on't  Send 


Figure  5.  Windows  XP  Error  Screen 
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Usability  Inspection 

Understanding  the  five  attributes  of  usable  systems  is  only  one  aspect  of  the 
usability  engineering  field.  Another  key  aspect  is  usability  inspection  -  the  quantifiable 
assessment  of  how  usable  a  system  actually  is.  Many  methods  exist  for  usability 
inspection.  Some  of  the  more  prevalent  of  these  methods  are  guideline  reviews, 
pluralistic  and  cognitive  walkthroughs,  consistency  and  standards  inspections,  formal 
usability  inspections,  feature  inspections,  and  heuristic  inspections.  Of  these,  the  most 
applicable  to  this  research  is  the  heuristic  usability  inspection  method. 

Heuristic  Inspections 

One  reason  the  heuristic  usability  inspection  method  is  most  applicable  to  this 
research  is  because  it  is  considered  a  “discount”  usability  inspection  method,  as 
referenced  in  the  literature  (Nielsen,  1994).  Such  a  term  is  applied  because  the  time, 
money,  and  resources  required  for  performing  this  method  are  low  compared  to  other 
methods  (Nielsen,  1994).  Given  the  exploratory  nature  of  this  research  though,  a 
heuristic  method  is  considered  ideal  since  the  goal  is  to  determine  broad  problem  areas. 
More  “detailed”  methods  can  then  be  used  to  conduct  more  in-depth  analysis  in  these 
areas. 

Another  reason  for  using  a  heuristic  method  is  because  Nielsen  (Useit.com,  1994) 
found  that  a  usability  engineering  intimidation  barrier  exists  in  most  organizations.  This 
barrier  results  from  the  perception  that  excessive  amounts  of  funding,  resources,  and  time 
are  necessary  for  implementing  usability  inspection  methods.  For  the  findings  and 
recommendations  of  this  research  to  be  accepted  and  integrated  into  the  Air  Force  Civil 
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Engineer  information  system  design  process,  it  will  be  necessary  to  break  through  this 
intimidation  barrier.  A  good  way  to  accomplish  this  is  through  the  use  of  usability 
engineering  methods  requiring  minimal  resources. 

One  factor  in  the  intimidation  barrier  is  the  complexity  of  usability-related  rules 
inherent  in  many  traditional  human-computer  interface  and  usability  design  techniques. 
These  techniques  can  often  exceed  a  thousand  rules  and  principles  in  their  methods  for 
evaluating  usability.  It  is  here  that  Nielsen  (Useit.com,  1994)  proposes  the  use  of  his  ten 
usability  principles  (as  outlined  in  the  following  sections)  in  place  of  the  thousands  of 
formal  rules.  The  usability  engineering  stands  a  much  better  chance  of  being  accepted  by 
an  organization  if  the  task  of  ensuring  usability  does  not  seem  as  formidable. 

Another  factor  is  the  cost  associated  with  deluxe  usability  techniques.  Typical 
cost  estimates  for  ensuring  usable  designs  using  formal  methods  are  on  the  order  of 
$60,000  or  more.  Nielsen  (Useit.com,  1994)  provides  evidence  that  good  value  usability 
engineering  practices  can  cost  six  times  less.  When  considered  in  relation  to  the 
hundreds  of  thousands  of  dollars  spent  on  any  given  information  system,  heuristic 
usability  engineering  costs  are  minor. 

Heuristic  methods  are  also  sometimes  perceived  as  less  effective.  Nielsen 
(Useit.com,  1994)  shows  that  the  benefit  to  cost  ratio  of  these  methods  are  consistently 
higher  than  the  ratio  provided  by  many  other  methods.  In  addition,  Nielsen  shows  that 
heuristic  methods,  while  they  may  not  find  every  single  usability  problem,  are  excellent 
for  detecting  the  major  usability  issues  in  a  given  information  system. 

Finally,  heuristic  methods  are  an  excellent  means  to  open  the  door  for  future, 
more  complex  usability  engineering  techniques.  Through  Nielsen’s  experience 
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(Useit.com,  1994),  he  has  learned  that  an  evolutionary  pattern  exists  for  usability 
engineering  in  most  organizations.  Such  an  evolution  is  rooted  in  validated  studies  of 
organizational  behavior  (Useit.com,  1994)  and  involves  starting  in  small  steps.  The  small 
steps  start  with  heuristic  techniques,  and  each  successive  iteration  convinces  more 
organizational  members  of  the  usability  engineering  benefits.  Gradually,  the  intimidation 
barrier  is  overcome  and  the  organization  fully  integrates  usability  engineering  into  its 
information  system  designs.  Thus,  the  key  to  acceptance  of  usability  engineering  lies  in 
starting  simple  and  adding  complexity  as  the  organizational  environment  allows. 

A  heuristic  inspection  involves  observing  an  information  system’s  adherence  to 
the  ten  heuristic  usability  principles  outlined  by  Nielsen  (1994):  visibility  of  system 
status;  match  between  the  system  and  real  world;  user  control  and  freedom;  consistency 
and  standards;  error  prevention;  recognition  rather  than  recall;  flexibility  and  efficiency 
of  use;  aesthetic  and  minimalist  design;  ability  to  help  users  recognize,  diagnose,  and 
recover  from  errors;  and  system  help  and  documentation.  Adherence  to  these  principles 
is  paramount  to  ensuring  that  the  learnability,  efficiency,  memorability,  low-error,  and 
user  satisfaction  requirements  of  usable  systems  are  met. 

Understanding  the  Heuristic  Principles 

Visibility  of  System  Status.  Visibility  of  system  status  describes  a  system’s 
ability  to  show  the  user  what  the  system  is  doing.  In  other  words,  an  IT  system  should  be 
letting  the  user  know  it  is  doing  something,  what  it  is  doing,  and  it  should  do  this  within  a 
reasonable  time  span.  Users  should  not  be  left  wondering  if  the  IT  system  has  crashed  or 
is  still  performing  an  operation  (Nielsen,  1994). 
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Match  Between  the  System  and  Real  World.  Match  between  the  system  and  real 
world  is  the  principle  that  requires  an  IT  system  to  use  terminology  familiar  to  the  users. 
Information  and  interaction  dialogues  created  by  the  system  should  be  in  terms  consistent 
with  the  real  world  and  familiar  to  the  people  who  will  use  the  system.  Furthermore, 
interactions  should  occur  in  a  natural  and  logical  order  (Nielsen,  1994). 

User  Control  and  Freedom.  A  usable  IT  system  should  allow  a  sufficient  amount 
of  user  control  and  freedom.  A  figurative  “backdoor”  should  be  available  at  any  time  in 
case  the  user  has  ventured  into  an  unintended  area  and  needs  to  back  out  of  the  unwanted 
transaction  and  return  to  desired  territory.  Navigation  buttons  such  as  “back”  and 
“forward,”  and  other  features  such  as  “redo”  and  “undo,”  are  examples  of  the  user  control 
and  freedom  principle  (Nielsen,  1994). 

Consistency  and  Standards.  Consistency  and  standards  are  necessary  to  ensure 
users  do  not  question  the  meaning  of  identical  icons  or  other  interactive  and  informative 
objects  used  in  different  contexts.  If  a  particular  symbol  has  a  particular  meaning  in  one 
area  of  the  IT  system,  it  should  have  the  same  meaning  in  all  other  areas.  A  prime 
example  of  this  is  the  “save”  icon  -  the  meaning  of  the  “save”  icon  is  universal  across  all 
Microsoft  Windows-based  applications  (Nielsen,  1994). 

Error  Prevention.  A  usable  system  should  have  a  sufficient  amount  of  error 
prevention.  This  principle  is  simple.  The  system  should  be  designed  such  that  is  does 
not  contain  errors,  and  the  system  should  not  lead  users  toward  committing  errors 
(Nielsen,  1994). 

Recognition  Rather  than  Recall.  Recognition  rather  than  recall  is  a  principle 
describing  a  system’s  ability  to  make  “objects,  actions,  and  options  visible”  (Nielsen, 
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1994:30).  The  purpose  of  objects,  actions,  and  options  ideally  should  be  intuitively 
obvious;  if  not,  clear  instructions  should  be  visibly  displayed  or  readily  available  through 
a  minimal  number  of  user  actions.  In  other  words,  if  a  dialogue  pops  up  on  a  user’s 
screen,  the  user  should  be  able  to  immediately  figure  out  the  purpose  and  usage  of  it; 
otherwise,  there  should  be  clear  instructions  about  its  purpose  and  the  actions  needed  to 
manipulate  it.  If  neither  of  these  is  possible,  the  user  should  be  able  to  access  a  help 
dialogue  with  a  minimal  number  of  keystrokes  or  mouse  clicks  (Nielsen,  1994). 

Flexibility  and  Efficiency  of  Use.  A  usable  IT  system  is  easy  enough  for  a  novice 
user  to  understand  and  operate,  yet  provides  expert  users  with  the  flexibility  and 
efficiency  of  use  derived  from  the  ability  to  custom-tailor  the  system  to  individual  user 
needs.  Once  a  novice  user  has  become  sufficiently  skilled  to  use  the  system  without  the 
need  for  deliberate,  step-by-step  functioning,  “accelerator”  options  should  be  available 
that  allow  the  user  to  access  frequently  used  features  more  quickly.  An  example  of  this 
might  be  the  capability  of  creating  macros  for  users  of  Microsoft  Word  -  expert  users  can 
condense  multiple  function-executing  keystrokes  or  icon  selections  into  a  single 
keystroke  (Nielsen,  1994). 

Aesthetic  and  Minimalist  Design.  Usable  IT  systems  should  be  constructed  in  an 
aesthetically  pleasing  and  minimalist  design.  Every  item  displayed  to  the  user  should  be 
attractive,  relevant,  and  as  brief  as  possible  so  as  not  to  waste  vital  screen  interface  space. 
Any  extraneous  items  compete  with  and  diminish  the  visibility  of  relevant  items  and 
should  be  avoided  (Nielsen,  1994). 

Help  Users  Recognize,  Diagnose,  and  Recover  from  Errors.  The  ability  of  an  IT 
system  to  help  users  recognize,  diagnose,  and  recover  from  errors  is  also  important  to 
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usability.  Although  errors  should  never  occur  under  the  “error  prevention”  principle  of 
usability  engineering,  when  errors  do  occur,  messages  about  the  error  should  be  displayed 
to  the  user  in  common,  understandable  terms.  Error  messages  should  refrain  from  using 
diagnostic  codes  to  describe  errors;  instead,  they  should  clearly  indicate  the  nature  of  the 
errors  and  suggest  relevant  and  constructive  solutions.  Users  should  not  need  to  consult  a 
system  administrator  or  system  developer  to  determine  the  solution  to  errors  that  occur 
(Nielsen,  1994). 

System  Help  and  Documentation.  System  help  and  documentation  should  ideally 
never  be  needed  in  a  fully  usable  system.  Since  this  is  rarely  the  case  though,  system 
help  and  documentation  should  be  easily  accessible.  Additionally,  it  should  be  as  concise 
as  possible,  relevant,  easily  searchable,  and  provide  step-by-step  instructions  so  users  are 
able  to  solve  specific  problems  (Nielsen,  1994). 

Measuring  the  Principles 

Understanding  the  principles  inherent  in  a  heuristic  inspection  is  important,  but  it 
raises  the  issue  of  how  such  principles  are  actually  measured.  Typically,  heuristic 
inspections  last  from  one  hour  to  half  a  day.  The  inspection  consists  of  an  evaluator 
going  through  the  pages  of  an  IT  system  and  determining  how  usable  the  system  is 
according  to  checklists  designed  to  measure  each  of  the  heuristic  principles.  Results  are 
aggregated  and  used  to  affect  necessary  changes  to  both  the  IT  system  and  the  associated 
iterative  design  process  (Nielsen,  1994). 

However,  it  should  be  emphasized  that  a  heuristic  inspection  is  an  evaluation  of 
the  information  system,  not  a  repair  of  it.  In  other  words,  a  heuristic  inspection  provides 
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solutions  in  terms  of  additional  insight  as  to  which  usability  principles  are  deficient  in  the 
system.  Heuristic  inspection  is  not  intended  to  provide  system  designers  with  software 
solutions  or  lines  of  machine  language  usability  code.  The  inspection  method  will, 
however,  provide  system  designers  and  program  managers  with  information  that  can  be 
used  to  improve  existing  design  practices  to  facilitate  usable  design  in  future  iterations  of 
the  system. 

Because  of  their  purpose,  heuristic  evaluations  may  not  provide  as  many 
indications  of  problems  as  more  complex  methods.  As  a  result,  heuristic  inspections  will 
certainly  not  find  every  single  problem  in  a  system.  They  will,  however,  find  more 
problems  than  doing  nothing  at  all  (Nielsen,  1994).  Furthermore,  while  heuristic 
inspection  methods  may  be  considered  “discount”  usability  engineering,  this  descriptor  is 
relative  to  other  inspection  methods  requiring  more  resources  (i.e.,  people,  time,  funding, 
and  equipment).  There  should  be  no  misunderstanding  that  heuristic  methods  still  require 
a  significant  amount  of  effort  on  the  part  of  the  evaluators  and  those  compiling  the  results 
(Nielsen,  1994).  However,  because  of  the  limited  number  of  inspectors  involved  in 
heuristic  inspections,  there  are  some  common  pitfalls:  biases  from  individual  inspectors, 
the  number  of  usability  problems  discovered  in  comparison  to  other  methods,  and  the 
inability  to  provide  specific  means  of  solution  (Nielsen,  1994). 

The  problems  associated  with  having  a  limited  number  of  inspectors  appear  to  be 
unpredictable.  Many  believe  that  proficient  inspectors  may  be  more  adept  at  problem¬ 
finding  than  others;  however,  some  research  has  shown  that  less  proficient  inspectors 
actually  identified  more  valid  problems  in  some  information  systems  (Nielsen,  1994).  In 
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other  words,  each  inspector  will  be  biased  in  some  fashion  to  some  degree,  and  this  bias 
may  manifest  itself  in  inspection  results. 

Existing  Air  Force  Standards 

While  understanding  the  basics  of  the  TAM  and  how  usability  engineering 
contributes  to  technology  acceptance,  the  regulations  and  guidance  governing  the  design 
of  Civil  Engineer  information  systems  are  what  ultimately  dictate  whether  or  not  usability 
engineering  principles  are  implemented.  Therefore,  this  section  provides  an  overview  of 
usability-related  Air  Force  standards  relevant  to  this  research.  Four  Air  Force  design 
frameworks  of  contemporary  significance  were  examined  in  the  course  of  this  research, 
as  well  as  more  specific  system-level  regulations  and  guidance  applicable  to  the 
information  system  used  in  the  methodology  of  this  research.  The  first  standard 
examined  was  the  Global  Combat  Support  System  (GCCS). 

The  Department  of  Defense  (DoD)  has  undertaken  efforts  to  integrate  the 
development  of  information  systems.  The  DoD  recognizes  that  a  wide  variety  of  stove- 
piped,  legacy  information  systems  currently  exist  within  its  organization,  each  designed 
differently  to  serve  different  needs.  Recent  efforts  have  been  undertaken  to  integrate  all 
stove-piped  DoD  combat  support  information  systems  into  a  single  system  that  can  be 
accessed  by  any  authorized  user  at  any  location  using  standardized,  commonly  available 
equipment.  The  hallmark  of  these  efforts  is  the  GCSS. 

Review  of  literature  available  on  the  GCSS  showed  that  this  framework  is 
intended  to  address  the  integration  issues  associated  with  combining  all  combat  support 
systems  into  a  universally  compatible  single  system.  A  focus  on  standardization  of 
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design  elements  from  a  usability  perspective  was  not  found.  Because  of  this,  as  well  as 
the  intended  focus  on  system-to-system,  rather  than  system-to-user  interface,  the  GCSS 
was  removed  from  consideration  as  an  applicable  framework  of  regulation  and  guidance 
for  information  system  design  in  a  usability  or  human-computer  interface  context.  The 
next  framework  examined  was  the  Technical  Architecture  Framework  for  Information 
Management  (TAFIM). 

The  TAFIM  was  found  to  be  comprised  of  several  volumes,  each  addressing 

different  aspects  of  information  system  design.  Of  these  volumes,  the  Human-Computer 

Interface  Style  Guide  (referred  to  in  this  research  as  simply  the  “TAFIM  Style  Guide”) 

was  the  only  one  containing  information  related  to  usability  engineering.  The  Style 

Guide  addresses  various  aspects  of  human-computer  interface,  including  hardware  and 

software  design.  Relevant  to  this  research,  the  software  specifications  consisted  mainly 

of  requirements  to  standardize  stylistic  design  features.  Such  standardization  of  system 

features  is  appropriate,  since  the  purpose  of  the  TAFIM  Style  Guide  (DoD,  1996)  is: 

“. .  .to  provide  a  common  framework  for  HCI  design  and 
implementation... interface  implementation  options  will  be  standardized, 
enabling  all  DoD  applications  to  appear  and  operate  in  a  reasonably 
consistent  manner... specifying  appearance,  operation,  and  behavior  of 
DoD  software  applications  will  support  the  following  operational 
objectives:  higher  productivity. .  .less  training  time. .  .reduced  development 
time.” 

Further  review  continued  to  show  a  primary  emphasis  on  standardizing  graphical  user 
interface  components  such  as  windows,  menus,  icons,  and  other  graphical  items  in  order 
to  provide  users  a  high  degree  of  transferability  of  component  meanings  across  different 
platforms.  The  TAFIM  Style  Guide  is  based  on  the  premise  that  standardization  will 
result  in  technology  acceptance:  “people  will  accept  and  use  what  is  easy  to  understand 
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if  it  aids  them  in  accomplishing  their  assigned  tasks  with  minimal  confusion  or 
frustration”  (DoD,  1996:  1-2).  Furthermore,  standardization  will  reduce  training  time 
because  users  do  not  need  to  re-leam  the  meaning  of  non- standardized  components: 
“standard  training  can  be  given  once  for  all  applications,  rather  than  requiring  users  be 
trained  when  transferring  to  new  systems”  (DoD,  1996:  1-2).  Finally,  the  guide  states 
that  system  development  time  is  reduced  because  system  components  are  standardized 
and  can  be  re-used  for  succeeding  systems  (DoD,  1996).  The  TAFIM  framework, 
including  the  Style  Guide,  was  considered  contemporary  until  1996,  when  it  was  replaced 
with  the  Joint  Technical  Architecture  (JTA). 

The  JTA  served  as  a  framework  similar  to  the  TAFIM  in  many  respects.  In 
relation  to  this  research,  the  key  differences  between  the  frameworks  was  the  inclusion  of 
commercial  standards  in  the  JTA  and  a  focus  on  maximizing  interoperability  between 
information  systems  and  technologies.  Per  the  DoD  (2003),  the  JTA  was  mandated  to 
only  include  a  minimum  set  of  standards  addressing  such  interoperability.  A 
comprehensive  review  of  JTA  standards  was  not  possible,  since  at  the  time  of  this 
research,  the  JTA  had  already  been  replaced  by  the  DoD  Information  Technology 
Standards  Registry  (DISR). 

The  DISR  is  a  set  of  standards  mandated  for  use  in  the  development  and 
acquisition  of  any  new  DoD  IT  system;  it  is  also  applicable  to  all  existing  DoD  systems. 
The  purpose  of  the  DISR  is  to  provide  a  “minimal  set  of  rules  governing  the  arrangement, 
interaction,  and  interdependence  of  system  parts  or  elements,  whose  purpose  is  to  ensure 
that  a  conformant  system  satisfies  a  specified  set  of  requirements”  (DoD,  2004: 15).  In 
other  words,  DISR  is  meant  to  ensure  the  interoperability  of  DoD  information  systems  by 
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establishing  a  set  of  baseline  standards  to  which  all  information  systems  must  adhere. 
While  this  may  sound  similar  to  the  purpose  of  the  JTA  and  the  TAFIM,  there  are  two 
key  differences.  First,  the  DISR  contains  only  those  standards  recommended  by  the 
Information  Technology  Standards  Committee  as  applicable  to  the  DoD  mission. 

Second,  the  DISR  is  comprised  of  an  Oracle  database  that  can  be  easily  queried  by  DoD 
employees  as  well  as  non-DoD  organizations  to  quickly  gain  access  to  standards. 

With  each  new  generation  of  regulations  and  guidance,  the  standards  have 
become  more  all-encompassing,  more  contemporary,  and  more  available  to  users.  The 
TAFIM  Style  Guide  was  originally  available  in  print  form,  and  consisted  of  several 
volumes  addressing  different  functional  standardization  needs.  The  JTA  expanded  on  the 
TAFIM  by  adding  a  wider  variety  of  commercial  standards.  The  DISR  created  an  easily 
searchable  online  standards  database  that  narrowed  the  standards  focus  to  those  standards 
recommended  by  the  Information  Technology  Standards  Committee,  a  neutral  committee 
of  information  technology  industry  experts  who  convene  to  agree  on  universal 
information  technology  standards  (Disronline.disa.mil,  2004). 

Examination  of  the  four  frameworks  (GCCS,  TAFIM,  JTA,  and  DISR)  was 
performed  to  provide  a  perspective  of  high-level  regulations  and  guidance  related  to  Civil 
Engineer  information  system  design.  Of  these  frameworks,  the  TAFIM,  JTA,  and  DISR 
were  found  to  be  relevant  to  this  research.  Although  the  TAFIM  and  JTA  are  superseded, 
they  are  relevant  for  the  following  reasons.  Information  systems,  and  more  importantly, 
the  information  system  used  in  the  methodology  of  this  research,  are  still  in  use  that  were 
constructed  to  the  TAFIM  Style  Guide  standard.  The  JTA  forms  the  main  body  of 
standards  contained  within  the  DISR  (the  most  recent  design  standard),  and  thus  bears  the 
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importance  of  being  mentioned.  Next,  an  examination  must  be  conducted  of  the  more 
detailed  specifications  pertaining  to  the  information  system  used  in  the  methodology  of 
this  research. 


The  Automated  Civil  Engineer  System  Personnel  Readiness  System 

As  will  be  outlined  in  Chapter  3,  the  information  system  chosen  as  the  unit  of 
analysis  in  this  research  is  the  Automated  Civil  Engineer  System  Personnel  Readiness 
(ACES-PR)  module.  Before  proceeding  any  further,  some  background  information  about 
ACES-PR  is  required. 

The  ACES-PR  module  is  an  information  system  designed  to  assist  Air  Force  Civil 
Engineer  Readiness  flights  in  managing  flight  operations  and  responsibilities.  More 
specifically: 


“personnel  training  and  readiness  equipment  management  require 
automated  information  system  (AIS)  support,  including  services  and 
preparations  for  wing  operations  during  natural  disasters,  major  accidents, 
war,  and  other  base  emergencies.  Applications  must  support  the  planning, 
programming,  and  training  for  the  protection  of  people,  resources,  and  the 
environment  from  the  effects  of  hazardous  explosive,  chemical, 
biological,  incendiary,  and  nuclear  ordnance.  In  addition,  the  AIS  must 
provide  for  the  management  of  information  and  resources  for  local  area 
and  areas  projected  for  deployment.  The  AIS  support  must  be  fully 
portable  in  support  of  deployments”  (AFCESA,  2003:7). 


It  is  established  in  the  ACES  Concept  of  Operations  (AFCESA,  2003)  that  all 
components  of  ACES  should  be  easily  manipulated  by  end  users.  The  ACES  information 
system  is  intended  to  “establish  a  user-friendly. .  .online  transactional  processing. .  .and 
online  analytical  processing  environment  for  Air  Force  warfighters”  (AFCESA, 
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2003:13),  while  modules  of  ACES  should  be  “engineered  to  provide  ‘common’  look  and 
feel  to  users  to  the  maximum  extent  possible  while  still  providing  a  desired  functional 
product”  (AFCESA,  2003:13). 

The  design  of  the  ACES-PR  module  was  guided  by  the  ACES-PR  Command, 
Control,  Communications,  Computers,  and  Intelligence  Support  Plan  (C4ISP)  (AFCESA, 
2003).  This  document  mandated  the  use  of  the  TAFIM  Style  Guide  for  all  design  of 
human-computer  interface  features  in  the  ACES-PR  system.  This  fact  is  relevant  because 
ACES-PR  was  designed  after  the  TAFIM  Style  Guide  had  already  been  superseded  by 
the  JTA. 

Another  standard  governing  the  design  of  the  ACES-PR  system  is  the 
Headquarters  Standard  Systems  Group  (HQ  SSG)  System  Engineering  Process  (SEP). 
The  SEP  provides  a  framework  for  system  planning,  analysis,  design,  implementation, 
maintenance,  and  closure;  its  existence  and  use  are  mandated  (DoD,  2004).  The 
overarching  purpose  of  the  SEP  is  to  “lay  out  a  plan  that  should  guide  all  technical 
aspects  of  an  acquisition  program”  (DoD,  2004:1).  The  core  documents  are  available  to 
designers  on  the  HQ  SSG  website  and  are  intended  to  be  custom-tailored  to  each 
application’s  development  process.  Customer  requirements  are  solicited  as  part  of  the 
SEP.  Surprisingly,  a  customer  must  express  usability  requirements  in  order  for  usability 
requirements  to  be  included  in  the  system  design  specifications.  In  other  words,  usability 
is  not  a  default  focus  of  the  SEP,  rather  it  must  be  explicitly  requested  by  customers. 
Various  sections  of  the  SEP  are  applicable  to  usability,  including  the  System  Design 
Document  and  the  Software  Test  Plan.  Below  are  reviews  of  those  sections  containing 
references  to  usability,  as  applied  to  the  ACES-PR  module  SEP. 
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The  ACES-PR  System  Design  Document  (SDD)  includes  the  technical  details  of 
the  design  process.  Its  purpose  is  to  describe  “the  structure  and  relationships  of  the  data 
required  to  manage  the  ACES-PR  system.  It  documents  the  design  decisions  that  will 
show  in  the  application  such  as  screen  shots  and  interface  requirements”  (HQ  SSG, 
2001(b):  1).  From  a  usability  perspective,  this  document  contains  the  main  specifications 
for  the  functional  operation  of  each  screen  that  the  user  sees.  In  terms  of  the  TAM,  the 
document  primarily  addresses  the  usefulness  of  ACES-PR  by  specifying  the  functional 
requirements  that  must  be  met  by  the  system. 

The  SEP  Software  Test  Plan  (STP)  template  specifies  several  types  of  testing. 
Integration,  interface,  or  interoperability  testing  measures  the  system’s  capability  of 
interacting  with  other  necessary  systems.  Stress  testing  involves  measuring  system 
capability  to  withstand  various  extremes  such  as  a  large  number  of  users  or  large 
numerical  input  values.  Performance  testing  focuses  on  network  throughput  capability, 
and  network  risk  assessment  deals  with  the  security  of  the  system  (HQ  SSG,  2001(a)). 
Similar  to  the  SEP,  customers  desiring  usability  testing  must  specifically  request  that  it  be 
included  in  the  software  test  plan. 

The  majority  of  the  tests  specified  in  the  ACES-PR  STP  are  functional  in  nature 
and  meant  to  evaluate  whether  or  not  the  system  produces  desired  outputs.  The 
components  of  the  STP  pertaining  to  usability  include  tests  to  measure  how  well  the 
system  meets  graphical  user  interface  standards  and  evaluate  error  messages.  The 
primary  methods  of  testing  include  entering  correct  and  incorrect  data  into  the  system  to 
see  if  the  system  responds  with  the  proper  output  for  a  given  input  and  to  determine  if 
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error  messages  are  meaningful  and  correct.  System  testing  is  performed  by  human  testers 
using  a  predetermined  list  of  test  actions. 

Chapter  Summary 

The  literature  review  was  necessary  to  gain  an  understanding  of  the  two  major 
fields  of  study  pertinent  to  this  research:  technology  acceptance  and  usability 
engineering.  The  heuristic  usability  inspection  method  was  reviewed  to  provide  a 
background  on  the  instrument  to  be  used  as  part  of  the  methodology.  An  examination  of 
the  standards  governing  the  usability  of  Civil  Engineer  information  systems  was 
conducted.  Finally,  an  understanding  of  the  ACES-PR  module  and  documents  relevant  to 
its  design  was  provided  to  familiarize  the  reader  with  the  unit  of  analysis  to  be  observed 
in  the  methodology  of  this  research. 
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III.  Methodology 


This  chapter  describes  the  methodology  used  in  this  research.  This  chapter  will 
include  a  discussion  of  the  different  research  methods  considered,  the  reasoning  behind 
choosing  the  case  study  method,  generalized  and  research-specific  descriptions  of  the 
initial  case  study  design  steps,  as  well  as  more  advanced  case  study  design,  protocol,  and 
data  gathering  procedures. 

Research  Methods 

The  scientific  method  involves  the  expansion  of  knowledge  through  the  use  of 
accepted  and  proven  research  methods.  This  research  is  no  exception  to  the  scientific 
method  and,  thus,  requires  the  selection  and  implementation  of  an  accepted  and  proven 
method  to  answer  the  research  questions.  Therefore,  before  focusing  on  the  methodology 
used  in  this  research,  a  broad  overview  will  be  provided  of  the  different  research  methods 
considered. 

Methods  Considered 

In  many  ways,  the  types  of  scientific  research  methods  available  can  be 
distinguished  by  their  levels  of  constraint.  In  the  field  of  scientific  research,  constraints 
are  defined  as  “restrictions  placed  on  the  researcher  in  an  effort  to  increase  the  precision 
of  the  research  and  enhance  the  validity  of  conclusions”  (Graziano  and  Raulin, 

2004:413).  Furthermore,  levels  of  constraint  are  the  degree  to  which  a  researcher 
“imposes  limits  or  controls  on  any  part  of  the  research  process”  (Graziano  and  Raulin, 
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2004:49).  Although  research  methods  are  often  labeled  as  either  being  high  or  low 
constraint,  the  levels  of  constraint  are  not  always  categorical.  For  a  given  research 
method,  the  levels  of  constraint  may  overlap  or  may  be  atypical  for  a  particular  type  of 
research.  For  example,  a  case  study,  typically  considered  a  low  constraint  form  of 
research,  may  in  some  cases  have  a  high  level  of  constraint. 

Levels  of  constraint  are  often  placed  on  what  is  termed  the  “unit  of  analysis.”  The 
unit  of  analysis  is  the  item  being  studied  and  analyzed  in  order  to  answer  the  research 
questions.  For  example,  if  a  researcher  was  trying  to  compare  the  standardized  test 
performance  of  two  demographics  of  people,  the  unit  of  analysis  would  be  the  individual 
person  belonging  to  a  specific  demographic  (Graziano  and  Raulin,  2004).  The  levels  of 
constraint  placed  on  the  unit  of  analysis  often  differentiate  types  of  research  methods. 

Five  types  of  research  methods  were  considered  and  are  listed  below  in  order  of 
increasing  levels  of  constraint.  Each  is  ordered  according  to  the  levels  of  constraint 
typically  assigned  to  that  particular  research  method. 

Naturalistic  Observation.  This  method  involves  observation  of  the  unit  of 
analysis  in  its  natural  environment.  Researchers  place  no  constraints  on  the  unit  of 
analysis  and  only  constrain  themselves  in  regards  to  interaction  with  the  unit  of  analysis. 
The  low  level  of  constraint  typical  of  naturalistic  observation  provides  the  benefit  of  high 
flexibility  in  observation.  If  the  behavior  of  the  unit  of  analysis  changes  during  the 
course  of  observation,  the  researcher  has  the  freedom  to  alter  research  strategies  to  best 
address  the  research  questions.  Research  questions  are  also  permitted  to  evolve  with  the 
situation.  Since  the  researcher  minimally  interferes  with  the  unit  of  analysis,  the  unit  of 
analysis  is  observed  in  its  natural  environment,  making  results  of  this  research  method 
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highly  applicable  in  the  real  world.  The  main  disadvantage  of  this  method  is  that  results 
are  not  given  as  much  credibility  as  a  high  constraint  method,  since  the  observation 
environment  is  not  strictly  controlled.  Although  there  are  exceptions,  the  naturalistic 
method  is  generally  regarded  by  scientists  as  exploratory  in  nature,  requiring  further 
explanatory  validation  (Graziano  and  Raulin,  2004). 

Case  Study  Method  of  Observation.  This  research  method  is  similar  to 
naturalistic  observation  in  that  the  researcher  tries  to  observe  naturalistic  behavior  of  the 
unit  of  analysis.  This  method  differs,  however,  in  that  the  researcher  is  given  more 
freedom  to  interact  with  the  unit  of  analysis.  Benefits  of  this  method  are  similar  to  the 
naturalistic  observation  method:  flexibility  in  observation  and  high  real-world 
applicability.  Disadvantages  are  also  similar,  as  case  studies  are  also  usually  regarded  as 
exploratory  (Graziano  and  Raulin,  2004). 

Correlational  Research.  The  correlational  research  method  involves  higher  levels 
of  constraint  than  case  studies  or  naturalistic  research.  The  goal  in  this  type  of  research  is 
to  observe  the  behavior  of  two  variables  to  discover  or  validate  a  correlation  between  the 
two.  The  higher  level  of  constraint  arises  from  the  need  to  strictly  control  the 
measurement  techniques  of  the  observers  to  ensure  consistent  observation  of  the 
variables.  The  advantage  of  this  method  is  that,  if  researchers  discover  a  correlation, 
results  allow  for  predicting  the  behavior  of  one  variable  based  on  the  behavior  of  the 
other.  The  tradeoff,  however,  is  that  researchers  lose  the  flexibility  of  changing 
observation  methods  to  evolving  research  questions.  Although  this  method  is 
explanatory,  it  does  not  prove  causality.  In  other  words,  correlational  methods  can  prove 
that  given  one  variable’s  behavior,  the  other  variable  will  behave  a  certain  way,  but  the 
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results  cannot  indicate  that  one  variable  actually  caused  the  other  variable’s  behavior 
(Graziano  and  Raulin,  2004). 

Differential  and  Experimental  Research.  Differential  and  experimental  research 
are  two  methods  in  which  high  levels  of  constraint  are  applied  to  the  observers,  the 
observation  methods,  and  the  units  of  analysis.  These  methods  involve  controlling 
groups  of  analysis  units  such  that  all  units  are  isolated  from  the  environment  in  order  to 
observe  specified  behaviors.  Observers  are  constrained  by  strictly  defined  observation 
methods  to  meet  the  goal  of  identical  observation  methods  across  all  units  and  groups  of 
units.  The  main  advantage  of  these  high  constraint  methods  is  that  results  are  regarded 
by  scientists  as  having  high  credibility  and  validity.  Differential  and  experimental 
research  methods  are  explanatory  and  may  prove  causality.  The  disadvantage  of  these 
methods  is  that  results  may  not  be  applicable  in  the  real  world.  Since  observation  takes 
place  in  an  isolated  environment,  the  environmental  factors  present  in  the  real  world  (and 
not  in  an  isolated  environment)  may  invalidate  results  (Graziano  and  Raulin,  2004). 

Method  Selected 

After  reviewing  the  research  methods  available,  the  case  study  research  method 
was  chosen.  The  reason  this  method  was  chosen  is  twofold:  the  levels  of  constraint 
applied  to  observers  needed  to  be  minimal  and  the  results  of  the  methodology  needed  to 
be  applicable  in  a  real-world  environment.  The  researcher  wanted  few  constraints  on 
observers.  In  order  to  make  pertinent  observations,  observers  needed  the  ability  to  freely 
interact  with  the  unit  of  analysis.  The  nature  of  the  unit  of  analysis  was  such  that 
behaviors  would  not  be  exhibited  without  observer  input.  In  addition,  observers  would 
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not  be  able  to  qualify  behaviors  without  the  freedom  to  vary  their  inputs  to  the  unit  of 
analysis.  Results  of  this  research  required  a  high  level  of  applicability  in  a  real-world 
setting.  Strictly  controlling  the  environment  in  which  the  unit  of  analysis  was  observed 
would  have  made  results  less  relevant,  since  the  unit  of  analysis  was  selected  as  an 
information  system  representative  of  other  systems.  Controlling  the  environment  of  the 
unit  of  analysis  would  have  made  the  system  less  representative.  Upon  choosing  a 
method,  it  was  necessary  to  define  the  layout  of  the  chosen  methodology.  To  accomplish 
this,  a  five-step  approach  was  used. 

Step  One:  Determine  Research  Questions 

The  initial  step  in  the  case  study  research  approach  was  to  determine  the  questions 
the  research  intended  to  address.  The  questions  needed  to  be  carefully  formulated  such 
that  their  nature  could  be  accurately  portrayed  in  the  form  of  who,  what,  when,  where, 
how,  or  why  (Yin,  2002). 

Step  Two:  Formulate  Research  Propositions 

The  second  step  in  the  case  study  research  approach  was  to  develop  propositions 
that  directed  “attention  to  something  that  should  be  examined  within  the  scope  of  study” 
(Yin,  2002:22).  This  step  forced  the  researcher  to  develop  propositions  about  what  the 
research  questions  implied.  While  the  research  questions  were  important,  the  substance 
of  the  research  was  defined  by  the  propositions.  As  applicable  to  this  research,  the 
following  propositions  were  made: 

1.  Usability  in  Civil  Engineer  information  systems  is  not  optimal. 
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2.  Non-optimal  usability  in  Civil  Engineer  information  systems  can  be  linked  to 
non- specification  of  usability  engineering  principles  in  Civil  Engineer 
information  system  regulations  and  guidance. 

Step  Three:  Determine  Unit  of  Analysis 

The  next  step  involved  .  .the  fundamental  problem  of  defining  what  the  ‘case’ 
is”  (Yin,  2002:22).  This  statement  implied  that  the  nature  of  the  unit  of  analysis  to  be 
studied  required  selection  in  a  fashion  that  allowed  the  research  questions  to  be  answered. 
However,  the  case  would  still  remain  flexible  and  its  definition  was  allowed  the  freedom 
to  change  as  relevant  observations  were  made  during  the  course  of  the  study  (Yin,  2002). 
The  research  questions  involved  characterizing  the  effects  of  specification  or  non¬ 
specification  of  usability  engineering  principles  by  Civil  Engineer  information  system 
regulations  and  guidance.  The  effects  were  to  be  qualified  by  observation  of  a  unit  of 
analysis  representative  of  Civil  Engineer  information  systems.  As  such,  the  ACES-PR 
module  was  chosen  as  the  unit  of  analysis. 

The  ACES-PR  module  can  be  considered  representative  of  other  Civil  Engineer 
information  systems  because  it  was  constructed  to  the  same  standards  as  other  systems. 
As  outlined  in  Chapter  2,  ACES-PR  was  designed  according  to  the  Headquarters  System 
Support  Group  system  engineering  process.  In  addition,  the  ACES-PR  module  design 
process  made  use  of  the  Technical  Architecture  Framework  for  Information 
Management,  and  continuing  design  iterations  are  subject  to  the  Department  of  Defense 
Information  Technology  Standards  Registry. 
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Step  Four:  Logic  Links  Data  to  Propositions 

The  fourth  step  in  the  case  study  research  approach  was  to  determine  the  logic 
that  would  link  the  research  data  to  the  propositions.  In  other  words,  how  would  the 
researcher  show  that  the  results  of  the  research  indicate  agreement  or  disagreement  with 
the  research  propositions  (Yin,  2002)?  As  applied  to  this  research,  the  research  data 
would  be  linked  to  the  propositions  as  outlined  in  the  “Case  Study  Protocol”  section  of 
this  chapter.  Usability  would  be  assessed  according  to  a  checklist  that  assessed  the 
usability  of  information  systems  according  to  the  principles  of  usability  engineering 
(Pierotti,  2002).  To  address  the  first  research  proposition,  the  case  study  unit  of  analysis 
(i.e.,  the  ACES-PR  module),  as  outlined  earlier,  would  be  considered  representative  of 
many  other  Civil  Engineer  information  systems,  since  most  Civil  Engineer  information 
systems  are  designed  to  the  same  set  of  standards.  To  address  the  second  research 
proposition,  data  gathered  during  the  research  would  be  referenced  back  to  existing  (in 
the  case  of  favorable  usability  measurements)  or  non-existing  (in  the  case  of  non- 
favorable  usability  measurements)  regulations  and  guidance  related  to  the  design  of  Civil 
Engineer  information  systems. 

Step  Five:  Criteria  for  Interpreting  Findings 

The  final  step  in  designing  the  case  study  was  defining  the  criteria  to  be  used  to 
interpret  the  research  findings.  The  rules  had  to  be  defined  that  determine  whether  the 
occurrence  of  a  particular  observation  indicated  a  particular  finding.  For  this  research, 
the  criteria  for  measuring  usability  were  provided  by  the  usability  checklist.  Checklist 
items  (observed  behaviors)  were  grouped  according  to  the  usability  engineering  principle 
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they  measured.  A  certain  number  of  usability-favorable  checklist  items  per  group  would 
yield  a  favorable  usability  result  for  the  respective  principle  (Pierotti,  2002).  More 
details  of  this  aspect  of  the  methodology  are  outlined  in  “Case  Study  Protocol”  section  of 
this  chapter,  as  well  as  in  Chapter  4  of  this  research. 

Detailed  Case  Study  Design 

After  the  initial  layout  of  the  case  study  approach  was  determined,  the  detailed 
case  study  design  had  to  be  accomplished.  The  following  sections  explain  the  detailed 
design  and  actual  procedures  of  the  case  study.  The  first  step  in  formulating  the  explicit 
design  of  the  case  study  was  to  determine  which  of  the  four  standard  design  types  to  use: 
single-case  (holistic),  single-case  (embedded),  multiple-case  (holistic),  or  multiple-case 
(embedded)  (Yin,  2002). 

The  literature  provided  several  scenarios  in  which  a  single-case  design  would  be 
appropriate.  The  scenario  applicable  to  this  research  was  to  choose  a  single-case  design 
if  the  unit  of  analysis  could  be  considered  representative  or  typical  of  many  similar  units 
of  analysis.  This  was  the  situation  with  this  research;  as  previously  stated,  the  ACES-PR 
system  was  considered  representative  of  other  systems  in  the  Civil  Engineer  career  field 
because  of  common  design  regulations  and  guidance. 

At  this  point,  whether  the  case  study  design  was  to  be  holistic  or  embedded  still 
had  to  be  determined.  A  holistic  design  approach  meant  using  a  single,  stand-alone  unit 
of  analysis  for  the  research,  whereas  an  embedded  design  would  have  involved  two  or 
more  of  the  same  types  of  analysis  units,  or  units  of  analysis  that  contained  subunits  (Yin, 
2002).  Though  the  goal  of  this  research  involved  making  generalizations  about  many 
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information  systems,  this  was  to  be  done  through  exploration  of  a  single  information 
system.  In  addition,  the  information  system  chosen  to  be  examined  was  not  comprised  of 
differentiable  subunits.  Since  the  target  of  this  research  was  a  single  information  system, 
the  ACES-PR  module,  and  this  information  system  contained  no  subunits,  the  holistic 
approach  was  more  appropriate.  An  embedded  approach  would  have  been  suitable  only 
if  the  research  methodology  was  to  be  executed  on  multiple  related  information  systems 
or  a  single  information  system  with  multiple  subsystems. 

Case  Study  Protocol 

Choosing  a  type  of  design  was  important,  but  just  as  important  was  the  design  of 
the  case  study  execution  itself  or  rather,  how  data  was  to  be  acquired  to  answer  the 
research  questions.  The  case  study  protocol  provided  the  framework  for  accomplishing 
this.  A  case  study  protocol  “contains  the  instrument  as  well  as  the  procedures  and 
general  rules  to  be  followed  in  using  the  protocol”  (Yin,  2002:67).  Its  purpose  is  to 
facilitate  the  research  by  providing  guidelines  for  data  collection  (Yin,  2002),  and  it 
should  contain  four  key  elements. 

1.  An  overview  of  the  case  study  explains  the  purpose  of  the  research  and 
provides  pertinent  background  information. 

2.  Field  procedures  represent  an  administrative  guide  to  prepare  the  researcher 
for  the  proper  collection  of  data. 

3.  Case  study  questions  include  not  only  the  research  questions,  but  specific 
questions  intended  to  elicit  appropriate  data. 

4.  A  case  study  report  guide  provides  instructions  on  how  to  report  the  results,  to 
include  format  and  presentation  requirements  (Yin,  2002). 
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The  specific  protocol  used  in  this  research  is  included  as  Appendix  A.  It  consists  of 
excerpts  from  the  first  and  second  chapters  of  this  research,  as  well  as  the  Pierotti  (2002) 
checklist  used  as  the  instrument  for  data  collection. 

The  checklist  designed  by  Pierotti  (2002)  is  intended  to  assess  the  usability  of 
information  systems.  The  checklist  contains  fourteen  sections  designed  to  measure  the 
usability  principles  of  visibility  of  system  status;  match  between  system  and  the  real 
world;  user  control  and  freedom;  consistency  and  standards;  error  prevention;  recognition 
rather  than  recall;  flexibility  and  efficiency  of  use;  aesthetic  and  minimalist  design; 
helping  users  recognize,  diagnose,  and  recover  from  errors;  help  and  documentation; 
flexibility  and  minimalist  design;  skills;  pleasurable  and  respectful  interaction  with  the 
user;  and  privacy  (Pierotti,  2002). 

Upon  initial  observation,  the  heuristic  principles  of  this  checklist  appear  to  be 
different  than  those  specified  by  Nielsen  (2004).  The  areas  in  Pierotti’ s  checklist  that 
differ  from  Nielsen’s  usability  principles  are:  flexibility  and  minimalist  design,  skills, 
pleasurable  and  respectful  interaction  with  the  user,  and  privacy.  Though  titled 
differently,  the  measured  areas  in  the  Pierotti  (2002)  checklist  adhere  to  the  general  idea 
of  a  heuristic  inspection  and  measure  essentially  the  same  characteristics  that  contribute 
to  the  usability  of  an  information  system.  The  only  exception  to  this  is  the  category  of 
privacy,  which  is  a  subject  area  not  covered  by  the  intentions  of  this  research.  As  a 
result,  the  privacy  measured  area  was  not  included  in  the  case  study  protocol  or  final 
results  of  this  research. 

The  choice  to  use  Pierotti’ s  (2002)  checklist  in  lieu  of  a  checklist  provided  by 
Nielsen  was  based  on  the  fact  that  Pierotti’ s  checklist  was  openly  available  at  no  cost  to 
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the  researcher.  Nielsen’s  checklist  was  available  for  purchase  at  a  rate  consistent  with 
Nielsen’s  fee  for  consultation  services.  Thus,  in  order  to  continue  the  purpose  of 
breaking  the  usability  engineering  organizational  intimidation  barrier,  it  was  appropriate 
to  choose  the  openly  available  checklist  although  it  did  not  share  the  exact  form  as 
Nielsen’s  usability  principles.  In  addition,  the  idea  of  altering  the  checklist  to  align  with 
Nielsen’s  principles  was  considered.  This  option  was  eliminated  for  two  reasons.  The 
first  reason  was  to  preserve  the  consistency  between  data  gathered  for  this  research  and 
data  gathered  by  other  researchers  using  the  Pierotti  (2002)  checklist.  The  second  reason 
was  to  ensure  the  measurement  instrument  remained  in  a  form  as  intended  by  Pierotti 
(2002).  Alterations  may  have  tainted  the  essence  of  each  measured  category  as  validated 
by  Pierotti. 

Each  checklist  item  was  answered  by  the  observers  on  a  “yes,  no,  or  not 
applicable”  scale.  The  “not  applicable”  rating  was  used  if  the  research  participant  did  not 
understand  the  checklist  item  or  believed  the  checklist  item  to  be  irrelevant  in  the  context 
of  the  observed  system.  In  addition,  each  item  on  the  checklist  provided  space  for 
observers  to  add  comments.  These  comments  were  provided  at  the  observers’  discretion 
to  qualify  responses  or  to  include  any  other  information  relevant  to  a  checklist  item. 

To  use  the  checklist  and  execute  the  protocol,  two  observers  were  chosen.  This 
number  of  observers  was  based  on  the  number  of  people  available  to  the  researcher  that 
possessed  enough  of  a  background  in  usability  engineering  and  information  systems  to 
understand  the  concepts  and  terminology  associated  with  the  case  study  instrument. 
Observer  1  was  a  web  administrator  with  one  year  of  experience  in  Windows-  and  Linux- 
based  website  and  graphical  design  using  multiple  programming  languages  including 
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HTML  and  PHP.  Observer  1  also  had  experience  with  content  management  systems  and 
database  management,  and  spent  two  years  as  a  workgroup  administrator.  Observer  1 
had  limited  experience  using  the  ACES-PR  module,  including  a  command- sponsored 
training  course  and  management  oversight  of  ACES-PR  end  users.  Observer  2  had  over 
19  years  total  experience  working  with  mainframe  and  desktop  computers.  In  the  last 
five  years,  Observer  2  had  acquired  software  programming  experience  in  the  Perl, 

HTML,  and  JavaScript  programming  languages  and  had  worked  as  a  project  manager  and 
system  administrator.  Observer  2  had  no  prior  experience  using  the  ACES-PR  module. 

Both  observers  had  research  experience  in  usability  engineering  and  technology 
acceptance  concepts.  In  addition,  both  were  familiar  with  the  case  study  methodology  as 
outlined  by  Yin  (2002).  Finally,  both  observers  were  graduate  students  enrolled  in 
information  technology  course  sequences  involving  the  study  of  contemporary  and 
historical  management  information  system  and  information  technology  topics.  The 
observers  executed  the  case  study  protocol  on  an  ACES-PR  module  demonstration 
database  and  not  an  active  database  containing  actual  user  data.  This  demonstration 
database  was  identical  to  the  fielded  system  except  that  the  data  was  fictional  rather  than 
actual. 
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Chapter  Summary 

This  chapter  outlined  the  general  approach  to  this  research  and  specifically 
explained  the  development  of  the  case  study  protocol  and  usability  checklist.  Discussion 
also  included  a  review  of  available  research  methods  and  a  description  of  the  rationale  for 
the  selected  methodology.  The  following  chapter  will  discuss  the  findings  of  the  case 
study  approach  used  in  this  research. 
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IV.  Results 


To  meet  the  original  research  objectives,  this  chapter  discusses  the  existing  level 
of  usability  standards  specified  for  the  design  of  a  representative  information  system,  the 
Automated  Civil  Engineer  Personnel  Readiness  (ACES-PR)  module.  The  majority  of  the 
chapter  provides  the  results  of  an  assessment  on  the  ACES-PR  module  to  qualify  the 
existence  of  any  possible  gaps  in  existing  standards. 

Existing  Usability  Standards 

Chapter  2  provided  a  discussion  of  the  general  contents  of  regulations  and 
guidance  associated  with  the  design  of  Civil  Engineer  information  systems.  Further 
discussion  of  these  standards  is  required  in  order  to  answer  the  first  research  question, 
“How  do  current  standards  related  to  the  design  of  Civil  Engineer  information  systems 
specify  usability  engineering  principles?”  The  standards  found  to  be  applicable  in 
answering  this  question  were  the  Joint  Technical  Architecture  (JTA),  Department  of 
Defense  (DoD)  Information  Technology  Standards  Registry  (DISR),  Technical 
Architecture  Framework  for  Information  Management  (TAFIM),  Headquarters  System 
Support  Group  (HQ  SSG)  System  Engineering  Process  (SEP),  and  the  Automated  Civil 
Engineer  System  Personnel  Readiness  (ACES-PR)  module  Software  Test  Plan  (STP).  As 
outlined  in  previous  chapters,  the  Global  Combat  Support  System  and  ACES-PR  System 
Design  Document  were  found  to  not  be  applicable  to  questions  about  usability 
engineering,  since  the  intent  of  these  documents  was  to  facilitate  integration  and 
usefulness  features  rather  than  usability  engineering. 
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As  stated  in  Chapter  2,  comprehensive  analysis  of  the  JTA  was  not  possible  since 
this  framework  had  already  been  absorbed  into  the  DISR  at  the  time  of  research.  As  such 
was  the  case,  the  DISR  was  examined  instead.  Within  the  DISR,  evidence  could  not  be 
found  of  usability-related  standards  related  to  general  information  system  design.  There 
were  standards  listed  in  the  registry  addressing  nuclear  weapon  human-computer 
interfaces  and  conventional  weapon  system  human-computer  interfaces 
(Disronline.disa.mil,  2004);  however,  no  standard  was  listed  for  general  information 
system  (non-nuclear,  non- weapon  systems)  human-computer  interfaces  or  usability 
engineering  in  any  other  form.  Thus,  in  the  case  of  currently  applicable  high-level 
information  system  design  guidance,  no  specification  existed  for  usability  engineering 
principles. 

A  previous  high-level  framework,  the  TAFIM,  contained  one  volume  pertinent  to 
the  discovery  of  usability  engineering  specifications.  This  volume,  the  TAFIM  Human- 
Computer  Interface  Style  Guide  (or  simply,  the  “TAFIM  Style  Guide”),  was  found  to 
contain  various  usability  specifications.  As  stated  in  Chapter  2,  these  specifications 
mainly  addressed  the  standardization  of  design  elements  across  all  systems  with  the  goals 
of  maximizing  usability  and  minimizing  required  user  learning  time.  Standardization  of 
design  elements  falls  under  the  “consistency  and  standards”  principle  of  usability 
engineering  (Nielsen,  1994).  No  specifications  applicable  to  the  other  usability 
engineering  principles  could  be  found  in  the  TAFIM  Style  Guide,  and  thus,  the 
conclusion  was  drawn  that  this  guide  only  partially  addressed  the  principles  of  usability 
engineering. 
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The  HQ  SSG,  responsible  for  the  development  of  a  wide  variety  of  Civil  Engineer 
information  systems,  including  the  ACES-PR  module,  mandates  the  use  of  its  SEP  for 
information  system  design.  Within  the  SEP,  there  exists  a  requirement  for  the  STP.  In 
the  realm  of  usability  specifications,  the  ACES-PR  STP  was  found  to  contain  tests  for 
compliance  with  graphical  user  interface  requirements  as  well  as  validity  and 
effectiveness  of  error  messages.  It  was  concluded  that  these  tests  pertained  to  the 
“consistency  and  standards”  and  “help  users  recognize,  diagnose  and  recover  from 
errors”  principles  of  usability  engineering.  No  specifications  applicable  to  the  other 
usability  engineering  principles  could  be  found  in  the  TAFIM  Style  Guide,  and  thus,  the 
conclusion  was  drawn  that  this  guide  only  partially  addressed  the  principles  of  usability 
engineering. 

Review  of  the  regulations  and  guidance  applicable  to  design  of  Civil  Engineer 
information  systems  indicated  that  in  some  cases,  applicable  usability  engineering 
standards  were  not  specified.  In  the  cases  where  usability-related  standards  were 
specified,  these  standards  did  not  address  all  the  usability  principles  as  recommended  by 
Nielsen  (1994).  Such  nonexistent  or  incomplete  specification  of  usability-related 
standards  validate  that,  indeed,  a  gap  exists  between  existing  standards  related  to  the 
design  of  Civil  Engineer  information  systems  and  an  optimal  usability  standard 
specifying  the  ten  principles  of  usability  engineering.  Upon  identification  of  a  usability 
gap,  it  is  pertinent  to  characterize  the  nature  of  the  nature  and  effects  of  such  a  gap. 
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Qualifying  Usability  Gaps 

The  second  research  question,  “How  are  gaps  in  existing  Civil  Engineer 
information  system  design  standards  qualified  upon  observation  of  Civil  Engineer 
information  systems  through  proven  and  accepted  usability  inspection  methods?”  dives 
further  into  exploring  the  gaps  revealed  by  the  first  research  question.  To  answer  the 
second  research  question,  a  representative  information  system  was  evaluated  using  the 
heuristic  usability  inspection  method.  The  rest  of  this  section  summarizes  the  results  of 
the  inspection  method  as  applied  to  the  ACES-PR  module.  The  results  are  first 
summarized  by  usability  principle  and  then  compiled  into  an  overall  usability  assessment 
based  on  the  results  of  the  individual  evaluations. 

For  each  usability  principle,  tables  are  presented  to  summarize  the  results  from 
applying  the  respective  portions  of  the  checklist  in  Appendix  A.  Each  table  contains  the 
individual  assessments  from  two  observers  as  well  as  the  total  assessment.  The  “not 
considered”  field  identifies  how  many  principle  evaluation  areas  were  considered  “not 
applicable”  by  either  or  both  observers.  The  “yes”  field  indicates  the  number  of 
observations  in  which  the  observer  agreed  that  the  information  system  met  the 
requirements  of  a  particular  evaluation  question.  Conversely,  the  “no”  field  contains  the 
number  of  observations  in  which  the  observer  did  not  believe  the  information  system  met 
the  requirements  of  a  particular  question.  The  “compliance”  field  is  the  percentage  of 
“yes”  responses  in  proportion  to  the  number  of  evaluation  areas  considered.  Finally,  the 
“average”  row  provides  the  overall  assessment  results  for  the  evaluated  area. 

The  compliance  percentage  was  compared  to  the  compliance  percentages 
categorized  by  Nielsen  and  Tahir  (2001).  A  compliance  rating  above  80%  indicates  “a 
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few  minor  fixes”  (Nielsen  and  Tahir,  2001:5)  to  the  system  are  required.  A  compliance 
rate  above  50%  and  below  80%  means  that  a  redesign  project  should  be  undertaken  to  fix 
isolated  usability  problems.  Below  50%,  the  compliance  rate  indicates  a  full  redesign  is 
necessary  and  implies  that  the  redesign  effort  should  more  effectively  address  the 
strategic  use  of  the  systems  as  well  as  the  users  and  their  needs  (Nielsen  and  Tahir,  2001). 

Assessment  Results  by  Usability  Principle  Group 

The  case  study  instrument  used  (Appendix  A)  was  a  heuristic  usability  checklist 
by  Pierotti  (2002).  As  described  in  Chapter  3,  the  Pierotti  (2002)  checklist  measures 
thirteen  usability  heuristics,  one  of  which,  the  privacy  heuristic,  was  excluded  from 
measurement  due  to  non-applicability  to  this  research.  Listed  below  are  the  results  of  the 
heuristic  inspection,  divided  into  individual  measured  areas. 

Visibility  of  System  Status 

The  visibility  of  system  status  is  the  usability  principle  describing  a  system’s 
ability  to  “keep  users  informed  about  what  is  going  on,  through  appropriate  feedback 
within  reasonable  time”  (Nielsen,  1994:30).  As  shown  in  Table  1,  The  ACES-PR 
systems  had  a  47.92%  compliance  rate,  which  was  a  very  poor  rating  in  comparison  to 
other  measured  areas.  Two  main  factors  contributed  to  this  rating:  significant  delays 
between  user  input  and  system  response  and  the  lack  of  accurate  and  consistent  visual 
feedback. 
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Table  1.  Visibility  of  System  Status 


Measured  Area: 

Visibility  of  System  Status 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

24 

5 

9 

15 

37.50% 

Observer  2: 

14 

10 

58.33% 

24 

5 

11.50 

12.50 

47.92% 

Observer  1  offered  a  significantly  lower  assessment  of  this  measured  area  than 
Observer  2.  This  may  have  been  due  to  biases  formed  through  prior  experience  with  the 
ACES-PR  module.  If  Observer  1  had  previous  negative  experiences  involving  this  area, 
the  observer’s  assessments  may  have  been  biased  toward  a  lower  assessment.  Individual 
differences  may  have  also  played  a  role.  If  Observer  1  had  a  lower  personal  perception 
of  what  was  an  appropriate  response  time  (i.e.,  was  less  patient)  than  Observer  2,  the 
assessment  of  Observer  1  may  have  been  more  likely  to  be  lower. 

Match  Between  the  System  and  Real  World 

The  principle,  match  between  system  and  real  world,  is  described  by  Nielsen 
(1994:30)  as  the  ability  of  a  system  to  “speak  the  users’  language,  with  words,  phrases, 
and  concepts  familiar  to  the  user,  rather  than  system-oriented  terms”  and  “follow  real- 
world  conventions,  making  information  appear  in  a  natural  and  logical  order.”  As  shown 
in  Table  2,  the  system  achieved  a  73.33%  compliance  rating  in  this  measured  area,  which 
is  considered  a  relatively  good  rating.  The  main  areas  requiring  improvement  included 
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the  non-standard  selection  of  function-indicative  colors  and  the  presence  of  similarly 
designed  icons  that  performed  opposite  or  distinctively  different  functions. 


Table  2.  Match  Between  the  System  and  Real  World 


Measured  Area: 

Match  Between  the  System  and  Real  World 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

15 

9 

8 

7 

53.33% 

Observer  2: 

14 

1 

93.33% 

15 

9 

11.00 

4.00 

73.33% 

Again,  Observer  1  offered  a  significantly  lower  assessment  of  the  system  than 
Observer  2.  As  with  the  last  measured  area,  this  may  have  been  due  in  part  to  pre-formed 
perceptions  about  the  system  as  a  result  of  having  prior  experience  with  ACES-PR.  In 
addition  to  this,  however,  Observer  1  was  very  familiar  with  the  real-world  functions  that 
this  system  supports,  while  Observer  2  had  little  or  no  knowledge  about  such  functions. 
As  a  result,  Observer  1  may  have  discovered  more  issues  in  this  area  because  of  a  more 
comprehensive  understanding  of  what  is  necessary  to  provide  a  match  between  this 
system  and  the  real  world  it  supports. 

User  Control  and  Freedom 

The  user  control  and  freedom  usability  principle  describes  a  system’s  ability  to 
provide  users  with  an  efficient  “escape  route”  in  the  event  that  the  user  operates  the 
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system  incorrectly.  This  may  be  caused  by  selecting  an  option  by  mistake  or  entering  the 


wrong  information.  In  either  case,  the  system  should  provide  “undo”  and  “redo” 
functions  (Nielsen,  1994).  The  system  performed  relatively  well  from  the  perspective  of 
the  user  control  and  freedom  principle,  with  a  70%  compliance  rate  as  shown  in  Table  3. 
The  most  prevalent  features  found  by  observers  to  be  missing  from  the  system  were  the 
ability  to  reverse  unwanted  actions  and  a  feature  to  cancel  out  of  operations  in  progress. 


Table  3.  User  Control  and  Freedom 


Measured  Area: 

User  Control  and  Freedom 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

15 

8 

11 

4 

73.33% 

Observer  2: 

10 

5 

66.67% 

15 

8 

10.50 

4.50 

70.00% 

Consistency  and  Standards 

The  consistency  and  standards  principle  is  intended  to  ensure  users  “should  not 
have  to  wonder  whether  different  words,  situations,  or  actions  mean  the  same  thing” 
(Nielsen,  1994:30).  A  system  adhering  to  this  principle  will  typically  use  standardized 
platform  conventions.  The  ACES-PR  systems  displayed  the  best  performance  in  regards 
to  consistency  and  standards  when  compared  to  other  measured  areas;  as  shown  in  Table 
4,  this  area  had  a  compliance  rating  of  87.88%.  The  main  issue  observed  under  this 
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principle  was  the  system’s  color  scheme.  For  one  observer,  colors  did  not  match  the 


color  schemes  that  the  observer  was  accustomed  to  seeing  in  other  information  systems. 

Observer  1  provided  a  significantly  lower  assessment  of  this  measured  area  than 
Observer  2.  As  with  other  measured  areas,  pre-formed  perceptions  due  to  previous 
experience  with  ACES-PR  may  have  affected  the  assessments  of  Observer  1.  In  addition, 
other  differences  may  have  been  due  to  the  information  technology  background 
differences  of  the  observers.  One  possible  factor  could  have  been  the  programming 
experience  of  the  observers.  Observer  2  had  experience  in  JavaScript  programming,  the 
language  that  forms  the  elements  of  ACES-PR.  With  the  perspective  of  a  programmer, 
Observer  2  may  have  had  a  different,  and  perhaps  more  tolerant,  perspective  on  what 
exactly  comprises  consistency  and  standards  in  a  JavaScript-based  information  system. 


Table  4.  Consistency  and  Standards 


Measured  Area: 

Consistency  and  Standards 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

33 

18 

25 

8 

75.76% 

Observer  2: 

33 

0 

100.00% 

BBS 

33 

18 

29.00 

4.00 

87.88% 

Help  Users  Recognize,  Diagnose,  and  Recover  from  Errors 

A  highly  usable  system  will  help  users  recognize,  diagnose,  and  recover  from 
errors.  As  stated  by  Nielsen  (1994:30),  “error  messages  should  be  expressed  in  plain 
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language  (no  codes),  precisely  indicate  the  problem,  and  constructively  suggest  a 
solution.”  The  ACES-PR  system  performed  well  in  this  area,  achieving  a  82.5% 
compliance  rating  as  shown  in  Table  5.  However,  the  observers  were  divided  in  their 
assessments;  one  observer  felt  the  system  was  primarily  in  compliance  while  the  other 
observed  some  discrepancies.  Observer  1  observed  that  the  system  error  messages  were 
less  directed  toward  user  resolution  of  the  error  and  more  directed  toward  informing  the 
user  that  an  error  had  indeed  occurred. 


Table  5.  Help  Users  Recognize,  Diagnose,  and  Recover  From  Errors 


Measured  Area: 

Help  Users  Recognize,  Diagnose,  and  Recover  from  Errors 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

20 

1 

14 

6 

70.00% 

Observer  2: 

19 

1 

95.00% 

20 

1 

16.50 

3.50 

82.50% 

Much  like  the  consistency  and  standards  assessment,  this  measured  area  was 
possibly  affected  by  pre-formed  perceptions  of  Observer  1,  and  the  more  extensive 
programming  experience  of  Observer  2.  Pre-formed  perceptions  would  explain  lower 
assessments  of  Observer  1.  Programming  experience  of  Observer  2  might  explain  a 
higher  assessment  by  Observer  2,  since  such  experience  might  make  an  observer  more 
tolerant  of  error  messages  using  the  system’s  and  not  users’  language. 
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Error  Prevention 


A  system  that  assists  users  in  recognizing,  diagnosing,  and  recovering  from  errors 
is  even  better  if  it  prevents  errors  from  occurring  in  the  first  place  (Nielsen,  1994).  This 
measured  area  was  observed  to  be  60%  in  compliance  as  shown  in  Table  6.  Menu  names 
were  found  to  differ  hierarchically;  in  other  words,  the  menu  option  name  differed  from 
the  name  displayed  on  the  function  that  operated  as  a  result  of  selecting  the  option. 
Additionally,  data  entry  fields  consistently  did  not  indicate  the  number  of  characters 
allowed  to  be  entered  into  the  field. 


Table  6.  Error  Prevention 


Measured  Area: 

Error  Prevention 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

10 

5 

7 

3 

70.00% 

Observer  2: 

5 

5 

50.00% 

BBS 

10 

5 

6.00 

4.00 

60.00% 

Recognition  Rather  than  Recall 

The  recognition  rather  than  recall  principle  relieves  the  user  of  the  burden  of 
having  to  “remember  information  from  one  part  of  the  dialogue  to  another”  (Nielsen, 
1994:30).  As  Table  7  indicates,  the  system  was  found  to  be  65%  in  compliance  with  the 
usability  heuristics  for  this  area.  One  reason  for  the  slightly  lower  rating  was  the 
placement  of  prompts,  cues,  and  messages  in  unexpected  areas,  i.e.,  places  on  the  screen 
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where  the  user  would  probably  not  be  looking.  Another  reason  was  the  minimal  use  of 


object  grouping  and  organizing  features  such  as  borders,  blank  spaces,  and  the  separation 
of  readable  chunks  in  long,  columnar  fields.  Data  entry  issues  also  included  vague 
marking  of  optional  and  dependent  entry  fields. 


Table  7.  Recognition  Rather  Than  Recall 


Measured  Area: 

Recognition  Rather  Than  Recall 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

30 

10 

17 

13 

56.67% 

Observer  2: 

22 

8 

73.33% 

Average: 

30 

10 

19.50 

10.50 

65.00% 

Pre-formed  perceptions  of  Observer  1  may  have  again  contributed  to  lower 
assessments  in  this  area;  however,  higher  assessments  of  Observer  2  may  have  resulted 
from  inexperience  with  the  system.  Observer  2,  being  inexperienced  in  the  use  of  ACES- 
PR,  may  not  have  been  aware  of  as  many  screens  available  to  users  as  Observer  1.  In 
other  words,  Observer  2  may  not  have  known  all  the  places  to  look  for  usability  problems 
and,  because  of  this,  may  not  have  accessed  the  features  that  indicated  to  Observer  1  that 
a  lower  assessment  was  required. 
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Flexibility  and  Minimalist  Design 

This  measured  area  is  described  by  Pierotti  (2002:7)  as  a  system’s  ability  to 
provide  “accelerators — unseen  by  the  novice  user — may  often  speed  up  the  interaction 
for  the  expert  user  such  that  the  system  can  cater  to  both  inexperienced  users  and 
experienced  users”  as  well  as  “alternative  means  of  access  and  operation  for  users  who 
differ  from  the  ‘average’  user.”  As  shown  in  Table  8,  the  system  was  assessed  as  having 
a  62.5%  usability  compliance  rate  in  this  measured  area. 


Table  8.  Flexibility  and  Minimalist  Design 


Measured  Area: 

Flexibility  and  Minimalist  Design 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

12 

4 

10 

2 

83.33% 

Observer  2: 

5 

7 

41.67% 

Average: 

12 

4 

7.50 

4.50 

62.50% 

The  main  finding  here  was  that  the  system  did  not  provide  many  features  that 
could  be  differentiated  with  respect  to  novice  or  expert  users.  There  was  a  single  level  of 
interface  language  applied  to  either  type  of  user.  The  higher  assessment  of  Observer  1 
may  have  been  due  to  the  observer’s  prior  experience  in  using  ACES-PR.  Having 
previously  used  ACES-PR,  Observer  1  may  have  formed  perceptions  about  what  types  of 
features  constitute  expert  or  novice  features  in  the  context  of  this  system.  In  addition, 
Observer  1  may  have  known  where  to  look  in  order  to  gain  access  to  such  features. 
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Observer  2  may  not  have  been  familiar  with  the  system  enough  to  know  of  features 
available  to  the  observer  for  customization. 

Aesthetic  and  Minimalist  Design 

Aesthetic  and  minimalist  design  describes  the  exclusion  of  unnecessary 
information  and  dialogues  (Nielsen,  1994).  The  system  scored  60%  compliance  in  this 
measured  area  as  shown  in  Table  9.  This  was  primarily  due  to  the  consistent  screen 
presence  of  information  unnecessary  to  the  decision  making  associated  with  the  screen, 
as  well  as  the  similarity  of  several  icons  that  were  conceptually  distinct. 


Table  9.  Aesthetic  and  Minimalist  Design 


Measured  Area: 

Aesthetic  and  Minimalist  Design 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

10 

2 

8 

2 

80.00% 

Observer  2: 

4 

6 

40.00% 

10 

2 

6.00 

4.00 

60.00% 

In  this  measured  area,  the  higher  measurements  of  Observer  1  were  probably  due 
to  the  observer’s  prior  experience  with  ACES-PR.  As  Observer  1  was  more  familiar  with 
the  system  than  Observer  2,  Observer  1  may  have  been  accustomed  to  the  appearance  of 
screen  elements  and  therefore,  not  as  negatively  affected  by  similarity  or  complexity  of 
screen  elements.  Individual  differences  may  have  played  a  role  as  well,  since  each 


65 


observer  was  likely  to  have  a  different  perception  of  what  constitutes  an  aesthetically 
pleasing  design. 

Help  and  Documentation 

System  help  and  documentation  should  be  easy  to  search  and  oriented  toward  user 
tasks;  it  should  specifically  list  step-by-step  instructions  and  be  concise  (Nielsen,  1994). 
The  system  complied  well  with  usability  principles,  scoring  a  73.81%  compliance  rating 
as  shown  in  Table  10.  Detractors  from  the  score  included  a  lack  of  navigation  and 
completion  instructions  on  data  entry  screens  and  other  dialogues.  There  was  also  a  lack 
of  explanatory  information  upon  selection  of  ambiguous  menu  items.  The  help  system 
interface  was  not  found  to  be  consistent  with  the  overall  system  interface;  it  differed  from 
the  rest  of  the  ACES-PR  system  in  format,  appearance,  navigation,  and  other  features. 


Table  10.  Help  and  Documentation 


Measured  Area: 

Help  and  Documentation 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

21 

2 

14 

7 

66.67% 

Observer  2: 

17 

4 

80.95% 

BBS 

21 

2 

15.50 

5.50 

73.81% 

Observer  1  may  have  provided  lower  results  again  based  on  pre-formed 
perceptions  due  to  previous  experience  with  ACES-PR.  Observer  2,  with  a  programming 
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background,  may  have  had  a  different  perception  of  what  is  required  to  provide  a  usable 
help  and  documentation  system.  Having  more  experience  in  programming  and 
information  technology  might  have  reduced  the  general  dependence  of  Observer  2  on 
help  and  documentation  features,  regardless  of  information  system  being  analyzed,  and 
because  of  this,  lowered  the  level  of  scrutiny  applied  to  this  element. 

Skills 

Pierotti  (2004:11)  describes  this  area  as  the  ability  of  a  system  to  “support, 
extend,  supplement,  or  enhance  the  user’s  skills,  background  knowledge,  and  expertise — 
not  replace  them.”  As  shown  in  Table  11,  the  system  was  assessed  as  having  a  60% 
usability  compliance  rating.  The  main  feature  in  this  category  that  was  not  observed  was 
the  ability  for  the  user  to  specify  iconic  or  textual  display  of  information.  In  addition,  the 
amount  of  information  displayed  per  screen  was  not  varied  in  response  to  a  user’s  skill 
level,  system  usage  frequency,  or  system  response  times. 


Table  11.  Skills 


Measured  Area: 

Skills 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

15 

6 

9 

6 

60.00% 

Observer  2: 

9 

6 

60.00% 

Average: 

15 

6 

9.00 

6.00 

60.00% 
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Pleasurable  and  Respectful  Interaction  with  the  User 

This  measured  area  describes  a  systems  ability  to  provide  users  with  enhancement 
to  their  work-life,  as  well  as  the  ability  of  the  system  to  treat  users  with  respect  (Pierotti, 
2002).  The  ACES-PR  module  was  assessed  relatively  high  in  this  category,  with  a 
usability  compliance  rating  of  78.57%  as  shown  in  Table  12.  Reasons  for  this  included 
an  icon  scheme  that  was  friendly  and  familiar  as  well  as  the  discretionary  use  of  color. 

Varying  observations  in  this  measured  area  were  likely  attributable  to  individual 
differences.  It  can  be  reasonably  assumed  that  each  observer  has  a  unique  perception  of 
what  can  be  considered  pleasurable  or  respectful  user  interaction.  In  addition,  the 
possibility  of  pre-formed  perceptions  existed  for  Observer  1  as  this  observer  had  previous 
experience  in  using  the  ACES-PR  module. 


Table  12.  Pleasurable  and  Respectful  Interaction  with  the  User 


Measured  Area: 

Pleasurable  and  Respectful  Interaction  with  the  User 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

7 

7 

4 

3 

57.14% 

Observer  2: 

7 

0 

100.00% 

Average: 

7 

7 

5.50 

1.50 

78.57% 

Assessment  Overall  Results 

The  final  overall  usability  assessment  was  determined  by  adding  the  average 
“yes”  responses  and  dividing  by  the  total  number  of  “considered”  responses.  The 
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average  “yes”  responses  refer  to  the  averaged  value  between  the  observers.  Overall,  the 
ACES-PR  system  was  considered  to  be  69.58%  compliant  with  heuristic  usability 
principles.  According  to  Nielsen  and  Tahir  (2001),  this  indicates  a  need  to  redesign 
isolated  parts  of  the  system  to  ensure  a  high  level  of  usability.  The  overall  results  are 
summarized  in  Table  13. 


Table  13.  Overall  Usability  Observations 


Overall 

Heuristic  Usability  Inspection  Results 

Considered 

Not 

Considered 

Yes 

No 

Compliance 

Observer  1: 

212 

77 

136 

76 

64.15% 

Observer  2: 

159 

53 

75.00% 

212 

77 

147.50 

64.50 

69.58% 

Consolidating  the  results  allows  a  better  view  of  which  areas,  in  particular,  were 
found  to  be  least  in  compliance  with  usability  heuristics.  Table  14  summarizes  the 
results,  ranking  the  measures  least  compliant  to  most  compliant.  The  results  of  the 
heuristic  inspection  indicate  that  the  ACES-PR  system  is  most  compliant  with  the 
“consistency  and  standards”  as  well  as  the  “help  users  recognize,  diagnose,  and  recover 
from  errors”  usability  principles.  Due  to  the  emphasis  of  Civil  Engineer  information 
system  regulations  and  guidance  on  these  usability  principles,  the  high  level  of 
compliance  in  these  areas  was  not  unexpected.  However,  the  observed  lack  of  design 
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standards  and  guidance  documents  specifying  the  inclusion  of  other  usability  principles 
may  have  contributed  to  lower  compliance  ratings  in  the  other  heuristic  inspection  areas. 


Table  14.  Summary  of  Heuristic  Inspection  Observations 


Considered 

Not 

Considered 

Yes 

No 

Compliance 

Visibility  of  System  Status 

24 

5 

11.50 

12.50 

47.92% 

Error  Prevention 

10 

5 

6.00 

4.00 

60.00% 

Aesthetic  and  Minimalist  Design 

10 

2 

6.00 

4.00 

60.00% 

Skills 

15 

6 

9.00 

6.00 

60.00% 

Flexibility  and  Minimalist  Design 

12 

4 

7.50 

4.50 

62.50% 

Recognition  Rather  Than  Recall 

30 

10 

19.50 

10.50 

65.00% 

User  Control  and  Freedom 

15 

8 

10.50 

4.50 

70.00% 

Match  Between  the  System  and 
Real  World 

15 

9 

11.00 

4.00 

73.33% 

Help  and  Documentation 

21 

2 

15.50 

5.50 

73.81% 

Pleasurable  and  Respectful 
Interaction  with  the  User 

7 

7 

5.50 

1.50 

78.57% 

Help  Users  Recognize,  Diagnose, 
and  Recover  from  Errors 

20 

1 

16.50 

3.50 

82.50% 

Consistency  and  Standards 

33 

18 

29.00 

4.00 

87.88% 

Thus,  to  answer  the  second  research  question,  the  gaps  in  existing  Civil  Engineer 
information  system  design  standards  (the  nature  of  which  were  determined  in  answering 
the  first  research  question)  are  qualified  as  corresponding  to  the  measured  usability 
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compliance  of  a  representative  information  system.  Low-measured  areas  in  the  ACES- 
PR  module  corresponded  to  usability  principles  not  specified  by  regulations  and 
guidance.  High-measured  areas  in  the  ACES-PR  module  corresponded  to  those 
principles  specified  by  regulations  and  guidance.  Furthermore,  given  the  representative 
nature  of  the  ACES-PR  module,  the  statement  can  be  made  that  all  Civil  Engineer 
information  systems  are  likely  to  exhibit  this  same  compliance  behavior.  What  can  be 
learned  from  this  finding  is  discussed  in  more  detail  in  the  following  section. 

Improvement 

Identifying  and  qualifying  gaps  in  usability  standards  allow  for  answering  the 
third  research  question,  “How  can  improvements  to  usability  be  made  as  a  result  of  any 
findings  yielded  by  the  usability  inspection  of  an  Civil  Engineer  information  system?” 
The  first  recommendation  for  improvement  is  to  address  the  root  cause  of  usability 
problems  and  not  necessarily  the  symptoms  alone.  The  research  results  are  general  in 
nature  and  the  intent  is  not  to  address  usability  problems  in  any  specific  system. 
Recommendations  will  then  be  provided  for  improving  usability  from  the  perspective  of 
the  proposed  root  cause,  the  regulations  and  guidance  governing  Civil  Engineer 
information  system  design. 

Addressing  the  Cause,  not  the  Symptoms 

The  results  presented  in  this  chapter  were  not  specific  in  nature.  That  is,  the 
researcher  did  not  provide  highly  detailed  references  to  causes  of  usability  problems 
within  the  evaluated  system;  instead,  general  assessments  of  usability  from  the 
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perspective  of  the  ten  usability  heuristics  (Nielsen,  1994)  were  provided.  To  further 
clarify,  outlining  a  specific  cause  of  usability  problems  might  be  to  point  out  the  need  for 
a  desirable  feature  such  as  a  specific  button  title  on  a  specific  page.  Such  specific  results 
were  not  provided  because  the  intent  of  this  research  was  not  to  point  out  specific  details 
in  any  given  information  system.  Instead,  the  goal  was  to  qualify,  categorically,  any  lack 
of  usability  regulations  or  guidance,  validate  the  effects  of  missing  regulations  or 
guidance  in  a  representative  Civil  Engineer  information  system,  and,  based  on  research 
results,  provide  recommendations  to  system  designers  and  program  managers  for 
improving  usability.  Pointing  out  specific  feature  issues  would  only  encourage  the  focus 
of  improvement  efforts  on  the  inspected  system’s  usability  problems,  when  the  results  of 
this  research  have  revealed  usability  issues  on  a  larger  scale.  Thus,  by  revealing  specific 
research  results,  the  researcher  would  be  promoting  efforts  toward  fixing  the  symptoms, 
and  not  the  cause,  of  usability  problems. 

Regulations  and  Guidance 

The  results  of  the  heuristic  usability  inspection  indicate  that  Civil  Engineer 
information  systems,  including  ACES-PR,  exhibit  usable  behavior  concurring  with  the 
level  of  specification  provided  in  applicable  regulations  and  guidance.  Because  of  this,  it 
needs  to  be  noted  that  these  systems  are  not  low  in  usability  compliance  due  to  violation 
of  Air  Force  standards,  but  rather,  it  is  the  standards  that  require  improvement. 

The  results  of  this  research  indicated  that  ACES-PR  exhibited  high  usability  from 
the  perspective  of  the  “consistency  and  standards”  usability  principle  as  this  measured 
area  was  shown  to  be  more  than  80%  compliant  with  heuristic  usability  measurements. 
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Such  a  high  rating  may  indicate  that  the  focus  of  regulations  and  guidance  on  consistency 
and  standards  in  the  design  of  Civil  Engineer  information  systems,  as  shown  in  the 
literature  review,  provided  the  ACES-PR  module  with  high  usability  in  this  area. 

With  the  exception  of  the  “help  users  recognize,  diagnose,  and  recover  from 
errors”  heuristic  principle,  all  other  usability  principles  measured  showed  low  ratings. 

The  high  (over  80%  compliance)  rating  of  the  “help  users  recognize,  diagnose,  and 
recover  from  errors”  measured  area  can  be  explained  by  specifications  for  error  message 
testing  in  the  ACES-PR  STP. 

One  recommendation  for  improvement  in  the  low-measured  usability  principles  is 
for  the  testing  and  evaluation  component  of  the  information  system  engineering  process 
to  include  a  usability  measurement  tool  such  as  the  checklist  used  in  the  case  study 
protocol  (Appendix  A)  of  this  research.  Contrary  to  the  methodology  of  this  research 
however,  very  specific  details  should  be  recorded  about  each  measured  area  for  use  by 
system  designers  in  rectifying  discovered  usability  issues.  Use  of  such  a  measurement 
tool  would  provide  designers  and  program  managers  with  valuable  feedback  on  a 
system’s  level  of  usability  while  still  in  the  design  stages,  allowing  changes  to  be  made 
before  the  information  system  is  fielded  to  end  users.  In  this  way,  designers,  and  not  end 
users,  discover  usability  issues.  As  a  result,  end  users  may  not  exhibit  low  technology 
acceptance  behavior  attributed  to  low  usability. 

A  regulation  governing  usable  information  system  design  should  be  included  in 
the  DISR.  As  previously  stated,  the  DISR  contains  regulations  for  nuclear  and  weapon 
system  design,  but  not  for  general  information  system  design.  The  TAFIM  Human 
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Computer  Interface  guide  could  be  updated  to  contain  usability  engineering,  and  then 
placed  in  the  DISR. 

Further  study  and  managerial  attention  should  be  focused  on  this  issue  to  provide 
a  more  widespread  perspective  on  the  detailed  issues  surrounding  usability  engineering  in 
the  Civil  Engineer  information  system  design  environment.  Undoubtedly  there  will  be 
resistive  forces  (Useit.com,  1994),  and  for  usability  engineering  to  maximize  technology 
acceptance,  such  resistance  will  need  to  be  addressed  from  the  highest  levels  of 
management. 

Chapter  Summary 

The  purpose  of  this  chapter  was  to  summarize  the  results  obtained  in  this  research. 
The  first  question  was,  “How  do  current  standards  related  to  the  design  of  Civil  Engineer 
information  systems  specify  usability  engineering  principles?”  This  was  answered 
through  further  examination  of  the  relevant  Air  Force  standards  found  in  Chapter  2.  The 
result  of  this  examination  showed  that  Air  Force  standards  in  some  cases  do  not  specify 
usability  engineering  principles  at  all,  and  in  other  cases  emphasize  the  “consistency  and 
standards”  and  “help  users  recognize,  diagnose,  and  recover  from  errors”  usability 
principles. 

The  second  question  was,  “How  are  gaps  in  existing  Civil  Engineer  information 
system  design  standards  qualified  upon  observation  of  Civil  Engineer  information 
systems  through  proven  and  accepted  usability  inspection  methods?”  By  evaluating  a 
representative  information  system,  it  was  shown  that  the  emphasis  of  Civil  Engineer 
standards  on  particular  usability  principles  was  reflected  in  current  information  systems. 
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The  results  of  the  evaluation  of  the  ACES-PR  module,  being  a  system  representative  of 
other  Civil  Engineer  information  systems,  indicate  that  gaps  in  existing  usability 
standards  contribute  to  low  compliance  with  heuristic  usability  principles. 

The  third  question  was  “How  can  improvements  to  usability  be  made  as  a  result 
of  any  findings  yielded  by  the  usability  inspection  of  a  Civil  Engineer  information 
system?”  This  question  was  answered  by  proposing  improvements  to  the  root  cause  of 
usability  problems,  the  regulations  and  guidance  documents,  and  the  nature  of  these 
proposed  improvements  was  characterized. 
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V.  Discussion 


While  Chapter  4  presented  results  and  recommendations  based  on  this  research, 
this  chapter  discusses  the  boundaries  of  this  research,  future  areas  of  study,  and  other 
research  efforts  related  to  this  research.  The  boundaries  are  explained  through  a 
discussion  of  limitations,  while  future  areas  of  study  are  those  suggested  to  further  the 
efforts  in  this  research.  Finally,  concurrent  research  efforts  by  other  Air  Force  Institute  of 
Technology  researchers  are  discussed. 

Limitations  of  Research 

As  with  any  research,  this  research  has  its  limitations.  The  main  limitations  of 
this  study  concern  its  scope  and  its  methods.  The  scope  of  this  research  was  limited  by 
manpower,  funding,  and  time. 

Manpower  limitations  were  one  factor  leading  to  the  selection  of  the  heuristic 
method  of  evaluating  usability  of  the  Automated  Civil  Engineer  System  Personnel 
Readiness  (ACES-PR)  module.  The  heuristic  method  is,  however,  a  valid  and  accepted 
method  of  assessing  usability,  and  was  suitable  for  the  purposes  of  the  case  study 
methodology  of  this  research.  Thus,  manpower’s  effect  as  a  limitation  in  conducting  the 
methodology,  from  this  perspective,  was  minimal. 

Funding  was  another  reason  for  choosing  the  heuristic  method,  since  only  a  few 
observers  are  required  to  accomplish  the  heuristic  method.  Other  methods  exist  for 
evaluating  usability,  but  these  methods  require  more  money  to  purchase  or  lease 
sophisticated  evaluation  hardware  and  software.  Again,  from  the  perspective  of 
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conducting  the  methodology  of  this  research,  the  effect  of  funding  as  a  limitation  on 
research  was  minimal. 

Of  more  significance  was  time.  Since  this  research  was  limited  to  18  months, 
explicit  results  yielded  from  recommendations  in  this  research,  and  thus,  practical 
validation  of  this  research,  have  yet  to  be  achieved.  Given  more  time,  more  complex  and 
widespread  research  could  have  been  conducted  on  more  information  systems,  in  more 
depth.  Results  of  a  longer  research  period  could  be  more  applicable  on  an  overall  Air 
Force- wide  scale,  as  discussed  further  in  this  section. 

Limitations  exist  in  the  methods  used  to  conduct  this  research.  These  limitations 
primarily  concerned  the  quantity  and  nature  of  observers  and  quantification  of  system 
user  perceptions.  A  limitation  in  the  heuristic  method  is  created  when  less  than  three 
evaluators  are  used  to  assess  a  system’s  usability.  According  to  Nielsen  (1994),  using 
two  evaluators  will  generally  only  discover  approximately  half  of  the  usability  problems 
in  any  given  evaluated  system.  Using  five  or  ten  evaluators  will  result  in  discovery  of 
approximately  75  percent  and  90  percent,  respectively,  of  usability  problems.  It  should 
be  noted,  however,  that  since  the  goals  of  this  research  were  exploratory  in  nature,  that 
the  effect  of  this  limitation  is  minimal.  The  purpose  was  not  to  detect  every  single 
usability  problem  in  a  particular  system;  instead,  the  goal  in  using  the  heuristic  method 
was  to  validate  the  existence  of  any  usability  issues  at  all.  Thus,  the  detection  of  usability 
issues  to  any  degree,  regardless  of  the  percentage  of  total  problems  found,  is  satisfactory 
in  meeting  the  goals  of  this  research. 

Another  limitation  of  the  observers  in  this  research  was  their  background.  As 
stated  in  Chapter  3,  Observer  1  had  mild  technical  experience  in  the  field  of  information 
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technologies,  as  well  as  previous  experience  using  and  managing  the  use  of  the  ACES-PR 
module.  The  extent  of  the  technical  knowledge  of  Observer  1,  as  well  as  prior  experience 
with  ACES-PR,  may  have  influenced  assessments  during  the  heuristic  inspection,  as 
outlined  in  depth  in  Chapter  4.  Observer  2  had  a  wealth  of  technical  experience, 
including  programming  experience,  but  had  no  prior  experience  with  ACES-PR.  Both 
conditions  may  have  affected  assessments  during  the  heuristic  inspection,  as  described  in 
Chapter  4.  Helping  to  offset  the  limitations  of  the  observer’s  backgrounds  was  their 
research  experience.  Comprised  of  studies  of  usability  engineering,  technology 
acceptance,  and  case  study  methodologies,  both  observers  were  aware  of  the  importance 
of  impartial  observations  in  performing  the  heuristic  inspection. 

This  research  could  have  benefited  from  an  initial  survey  of  ACES-PR  end  users. 
The  initial  proposal  that  a  low  usability  issue  existed  was  made  based  on  archival, 
personal,  and  anecdotal  evidence.  A  survey  to  end  users,  consisting  of  questions 
designed  to  assess  the  users’  perspective  on  usability  levels  of,  and  satisfaction  in  using 
the  ACES-PR  module,  could  have  helped  explicitly  determine  an  initial  level  of 
technology  acceptance.  This  same  survey  could  then  be  administered  at  a  later  point  in 
time  to  quantify  any  increases  in  the  level  of  technology  acceptance  resulting  from 
system  engineering  process  improvements  and  ultimately,  system  usability 
improvements.  Such  a  survey  could  serve  to  validate  the  results  and  importance  of  this 
research. 
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Suggested  Future  Research 

The  results  of  this  research  are,  by  no  means,  the  final  step  in  researching  Civil 
Engineer  information  system  usability.  The  results  of  this  research  are  only  a  minute  step 
toward  further  usability  research  in  the  Air  Force,  DoD,  and  other  federal  agencies.  This 
section  will  summarize  the  researcher’s  thoughts  on  ideas  for  future  research. 

The  research  results  could  be  further  validated  by  more  studies  similar  in  nature, 
but  using  other  information  systems  as  analysis  units.  Future  researchers,  perhaps  with 
career  field  backgrounds  besides  Air  Force  Civil  Engineering,  could  perform  an  identical 
holistic  single-case  study  using  another  information  system  that  is  relevant  to  their  career 
field,  or  perhaps,  to  remove  any  researcher  bias,  perform  the  study  on  an  information 
system  in  which  they  have  no  previous  experience.  Another  option  could  be  to  perform  a 
similar  study  using  a  different  approach  such  as  an  embedded  multiple  case  study,  in 
which  the  results  of  this  research  could  be  used  as  one  of  the  cases.  Such  a  methodology 
would  provide  improved  research  credibility  through  revelation  (or  non-revelation)  of 
regulation-  and  guidance-rooted  usability  issues  across  multiple,  unrelated,  yet 
representative,  information  system  platforms. 

Future  research  could  also  be  performed  on  a  larger  scale.  This  research  focused 
on  Civil  Engineer  information  systems,  but  the  issue  of  usability  engineering  can  be 
applied  at  higher  levels  such  as  the  Air  Force,  Department  of  Defense,  or  even  the  entire 
United  States  federal  government.  A  foreseeable  issue  would  be  finding  a  single 
information  system  representative  of  all  information  systems  contained  in  the  scope  of 
the  proposed  research.  In  this  case,  a  multiple-case  study  methodology  involving 
information  systems  from  various  government  agencies  might  be  appropriate. 
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Undoubtedly  there  will  be  resistance  to  usability  improvement  efforts.  To  assist 
in  dealing  with  such  resistance  to  change,  research  should  be  conducted  to  identify  the 
key  factors  contributing  to  and  mitigating  resistance  to  usability  improvement  efforts. 
Usability  engineering,  if  not  already  a  part  of  the  system  engineering  process,  will  require 
time,  manpower,  and  resources.  To  address  time,  manpower,  and  resource  problems,  it 
may  be  beneficial  to  conduct  a  cost  and  benefits  analysis  of  improving  usability  to 
quantify  tangible  effects  of  making  usability  improvements.  Such  an  analysis  might  be 
directed  toward  quantifying  the  effects  of  low  technology  acceptance  as  the  result  of  poor 
usability.  As  stated  in  previous  chapters,  there  is  a  correlation  between  usability  and 
technology  acceptance  that  has  been  repeatedly  validated,  and  analysis  of  the  effects  of 
low  technology  acceptance  might  reduce  managerial  resistance  to  directing  effort  toward 
usability  engineering.  Additionally,  it  is  recommended  that  any  efforts  to  overcome 
organizational  resistance  begin  with  heuristic  usability  engineering  methods.  As  stated  in 
Chapter  2,  such  methods  help  to  ease  the  process  of  breaking  through  the  usability 
intimidation  barrier  (Useit.com,  1994). 

Although  the  use  of  heuristic  methods  is  prescribed  for  initial  usability  efforts,  the 
benefits  of  more  formal  methods  are  also  great.  Because  of  this,  it  may  be  beneficial  to 
perform  research  using  other  inspection  methods  aside  from  the  heuristic  method  used  in 
this  research.  Many  more  costly  methods  are  available  to  researchers  (Nielsen,  1994), 
but  if  a  sponsor  can  provide  time,  manpower,  and  resources  to  a  future  researcher,  these 
methods  have  many  benefits  that  can  contribute  to  research  validity  and  credibility. 

Training  and  education  of  system  developers  and  program  managers  was  not 
addressed  in  this  research.  Future  research  should  examine  the  training  and  education 
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process  to  determine  the  level  of  usability  engineering  taught  to  people  associated  with 
the  information  system  engineering  process.  If  the  level  of  usability  engineering  training 
is  minimal,  it  may  be  beneficial  to  study  the  potential  benefits  of  exposing  system 
designers  and  program  managers  to  usability  engineering  concepts  and  practices. 

Air  Force  usability  engineering  efforts  could  benefit  from  a  research  duration 
longer  than  that  of  this  research.  A  longitudinal  study  of  information  system  usability 
would  add  more  weight  to  the  argument  for  usability  engineering  practices.  By 
performing  the  same  methodology  at  successive  points  in  time,  it  could  be  demonstrated 
that  low  usability  is  consistently  a  behavior  exhibited  by  organizational  information 
systems.  Another  benefit  of  taking  several  snapshots  in  time  is  that  program  managers 
and  decision  makers  would  receive  feedback  on  the  results  of  their  system  engineering 
process  improvements. 

Concurrent  Research 

For  reference  purposes  and  to  benefit  future  researchers,  it  is  important  to  note 
that  several  other  related  research  efforts  were  underway  at  the  Air  Force  Institute  of 
Technology  during  the  execution  of  this  research.  One  study,  involving  technology 
acceptance,  applied  the  Davis  et  al.  (1989)  model  of  technology  acceptance  to  the 
Communities  of  Practice  concept  implemented  in  the  Air  Force  Knowledge  Now  website. 
Also  involving  the  Air  Force  Knowledge  Now  website,  a  second  study  was  underway 
that  examined  the  website’s  usability  and  accessibility. 

The  first  study,  applying  the  Technology  Acceptance  Model,  is  relevant  because  it 
examined  the  factors  in  an  Air  Force  information  system  that  contributed  to  its 
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acceptance  or  non-acceptance.  The  findings  of  this  study  can  be  referenced  by  future 
researchers  to  assist  in  developing  their  research  of  usability  engineering,  since  ease  of 
use  is  a  key  factor  in  technology  acceptance.  This  research  thesis,  by  1st  Lt  John  Tate, 
was  to  be  completed  in  March  2005. 

The  second  study,  involving  usability  and  accessibility,  is  particularly  applicable 
to  usability  engineering  researchers  expanding  on  this  research  because  it  includes  the  use 
of  the  heuristic  inspection  method  as  well  as  Yin  (2002)  case  study  methodology.  This 
study  also  looks  at  accessibility,  a  factor  of  usability  not  addressed  by  Nielsen’s  (1994) 
ten  heuristic  principles.  Accessibility  is  important  because  it  ensures  that,  through 
assistive  technologies,  people  with  disabilities  are  able  to  use  information  systems.  This 
research  thesis,  by  Capt  Gary  Felax,  was  to  be  completed  in  March  2005. 

Chapter  Overview 

This  chapter  discussed  the  results  summarized  in  earlier  chapters.  The  boundaries 
of  this  research  were  explained  in  a  discussion  of  limitations.  Several  ideas  were 
described  for  future  researchers  to  use  in  efforts  to  explore  new  areas  of  research  and  to 
validate  and  use  this  research.  Finally,  a  reference  list  was  provided  of  related  research 
efforts  underway  at  the  time  of  this  research  to  help  future  researchers  in  their  efforts  to 
study  usability  engineering  and  technology  acceptance. 
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Appendix  A:  Case  Study  Protocol 


Overview 

The  purpose  of  this  case  study  instrument  is  to  facilitate  the  gathering  of  data  in 
order  to  evaluate  the  usability  of  the  Automated  Civil  Engineer  System  Personnel 
Readiness  (ACES-PR)  Module. 

The  ACES-PR  Module  is  a  web-based  information  system  used  by  Air  Force 
Civil  Engineer  personnel  to  manage  data  related  to  the  disaster  preparedness  and  Civil 
Engineer  world-wide  mobility  mission  areas. 

The  goal  of  the  research  observer  is  to  collect  information  about  the  usability  of 
the  ACES-PR  Module  using  the  case  study  instrument,  the  heuristic  inspection  checklist. 

Field  Procedures 

The  computers  available  for  evaluating  the  ACES-PR  module  are  located  in 
Wright-Patterson  Air  Force  Base,  Area  B,  Building  643  (The  Civil  Engineer  and  Services 
School),  Computer  Room  227.  These  computers  may  be  accessed  using  a  standard  Air 
Force  Institute  of  Technology  (AFIT)  student  user  account.  The  availability  of  this 
computer  room  is  listed  in  the  AFIT  email  Public  Folders,  accessible  from  the  highlighted 
Microsoft  Outlook  calendar  as  shown  in  the  Figure  on  the  next  page.  Any  block  of  time 
without  a  class  or  other  event  scheduled  for  the  room  can  be  considered  as  available  for 
use  by  case  study  observers. 

Once  logged  onto  an  evaluation  computer,  proceed  to  the  following  website  using 
Microsoft  Internet  Explorer  web  browser: 
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https  ://gupe64501.mont.disa.mil/servlet/f60servlet?config=aces 


The  web  browser  will  load  a  login  page.  The  user  name,  password,  and  database 
will  be  provided  to  you  by  the  researcher,  Capt  Kastenholz.  Submit  the  password  form 
and  the  browser  will  load  an  initial  page  providing  access  to  several  ACES  modules. 
Click  the  link  button  referencing  ACES-PR  (Personnel  Readiness).  Another  page  will 
load  which  is  the  initial  interface  for  using  all  features  of  ACES-PR.  At  this  point  the 
heuristic  evaluation  begins  and  the  case  study  observer  is  to  execute  the  case  study 
instrument,  the  heuristic  checklist. 


Folder  List 

0  Public  Folders 
Favorites 

0  <2-1  All  Public  Folders 

IS  ysb  ACSC/OL-A  Air  Command  and  Staf 
0  yt-l  AFIADL/DB  Air  Technology  Networ 
0  <2b  AFIT-ASA-TEXTBOOK  EXCHANGE 
<2-1  All  Public  Folders 
<2-1  BLD  470  classroom  schedule 
0  ^J-l  CC  AFIT  Command  Section 
0  ®  CE  Civil  Engineer  &  Services  Schoo 
0  Admin 

0  0  AFIT  Former  Staff 
0  AFIT/CE  IMAs 
0  ^2b  Bldg  643  Rooms 
0  Atrium 
0  Auditorium 
0  Computer  Room  221 
0  Computer  Room  227 

Figure.  Computer  Room  Availability 

Any  questions  related  to  execution  of  this  case  study  instrument  should  be 
directed  to  the  researcher,  Capt  Gunther  Kastenholz,  at  (937)  475-9631  or  via  email  at 
gunther  .kastenholz  @  afit.edu . 
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Case  Study  Instrument  (Heuristic  Checklist) 

The  case  study  observer  will  complete  the  following  checklist.  The  observer 
should  answer  each  question  by  filling  in  the  appropriate  circle  for  a  “yes,”  “no,”  or  “not 
applicable”  response,  as  indicated  in  the  checklist  by  “Y,”  “N,”  or  “NA,”  respectively.  A 
“yes”  answer  indicates  that  the  observer  agrees  that  the  ACES-PR  module  generally 
behaves  in  accordance  with  the  question.  A  “no”  answer  indicates  the  observer  does  not 
agree  that  the  ACES-PR  module  generally  behaves  in  accordance  with  the  question.  A 
“not  applicable”  response  indicates  that  the  observer  either  does  not  feel  the  question  is 
applicable  to  the  ACES-PR  module,  the  observer  feels  the  question  is  unclear,  or  the 
observer  otherwise  does  not  feel  a  “yes”  or  “no”  answer  is  justified.  The  observer  should 
also  add  any  appropriate  comments  that  the  observer  feels  would  provide  value  to  the 
researcher  in  understanding  the  reasoning  behind  choosing  the  selected  answer. 
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1.  Visibility  of  System  Status 

The  system  should  always  keep  user  informed  about  what  is  going  on,  through  appropriate  feedback  within 
reasonable  time. 


# 

Review  Checklist 

Y  NNA 

Comments 

1.1 

Does  every  display  begin  with  a  title  or 
header  that  describes  screen  contents? 

ooo 

1.2 

Is  there  a  consistent  icon  design  scheme 
and  stylistic  treatment  across  the  system? 

ooo 

1.3 

Is  a  single,  selected  icon  clearly  visible 
when  surrounded  by  unselected  icons? 

ooo 

1.4 

Do  menu  instructions,  prompts,  and  error 
messages  appear  in  the  same  place(s)  on 
each  menu? 

ooo 

1.5 

In  multipage  data  entry  screens,  is  each 
page  labeled  to  show  its  relation  to 
others? 

ooo 

1.6 

If  overtype  and  insert  mode  are  both 
available,  is  there  a  visible  indication  of 
which  one  the  user  is  in? 

ooo 

1.7 

If  pop-up  windows  are  used  to  display 
error  messages,  do  they  allow  the  user  to 
see  the  field  in  error? 

ooo 

1.8 

Is  there  some  form  of  system  feedback  for 
every  operator  action? 

ooo 

1.9 

After  the  user  completes  an  action  (or 
group  of  actions),  does  the  feedback 
indicate  that  the  next  group  of  actions  can 
be  started? 

ooo 

1.1 

Is  there  visual  feedback  in  menus  or 
dialog  boxes  about  which  choices  are 
selectable? 

ooo 
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1.11 

Is  there  visual  feedback  in  menus  or 
dialog  boxes  about  which  choice  the 
cursor  is  on  now? 

OOO 

1.12 

If  multiple  options  can  be  selected  in  a 
menu  or  dialog  box,  is  there  visual 
feedback  about  which  options  are  already 
selected? 

OOO 

1.13 

Is  there  visual  feedback  when  objects  are 
selected  or  moved? 

OOO 

1.14 

Is  the  current  status  of  an  icon  clearly 
indicated? 

OOO 

1.15 

Is  there  feedback  when  function  keys  are 
pressed? 

OOO 

1.16 

If  there  are  observable  delays  (greater 
than  fifteen  seconds)  in  the  system’s 
response  time,  is  the  user  kept  informed 
of  the  system's  progress? 

OOO 

1.17 

Are  response  times  appropriate  to  the 
task? 

OOO 

1.18 

Typing,  cursor  motion,  mouse  selection: 
50-1  50  milliseconds 

OOO 

1.19 

Simple,  frequent  tasks:  less  than  1  second 

OOO 

1.2 

Common  tasks:  2-4  seconds 

OOO 

1.21 

Complex  tasks:  8-12  seconds 

OOO 

1.22 

Are  response  times  appropriate  to  the 
user's  cognitive  processing? 

OOO 
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1.23 

Continuity  of  thinking  is  required  and 
information  must  be  remembered 
throughout  several  responses:  less  than 
two  seconds. 

OOO 

1.24 

High  levels  of  concentration  aren't 
necessary  and  remembering  information 
is  not  required:  two  to  fifteen  seconds. 

OOO 

1.25 

Is  the  menu-naming  terminology 
consistent  with  the  user's  task  domain? 

OOO 

1.26 

Does  the  system  provide  visibility:  that  is, 
by  looking,  can  the  user  tell  the  state  of 
the  system  and  the  alternatives  for  action? 

OOO 

1.27 

Do  GUI  menus  make  obvious  which  item 
has  been  selected? 

OOO 

1.28 

Do  GUI  menus  make  obvious  whether 
deselection  is  possible? 

OOO 

1.29 

If  users  must  navigate  between  multiple 
screens,  does  the  system  use  context 
labels,  menu  maps,  and  place  markers  as 
navigational  aids? 

OOO 

2.  Match  Between  System  and  the  Real  World 


The  system  should  speak  the  user’s  language,  with  words,  phrases  and  concepts  familiar  to  the  user,  rather 
than  system-oriented  terms.  Follow  real-world  conventions,  making  information  appear  in  a  natural  and 
logical  order. 


# 

Review  Checklist 

Y  NNA 

Comments 

2.1 

Are  icons  concrete  and  familiar? 

OOO 

2.2 

Are  menu  choices  ordered  in  the  most 
logical  way,  given  the  user,  the  item 
names,  and  the  task  variables? 

OOO 

2.3 

If  there  is  a  natural  sequence  to  menu 
choices,  has  it  been  used? 

OOO 
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2.4 

Do  related  and  interdependent  fields 
appear  on  the  same  screen? 

OOO 

2.5 

If  shape  is  used  as  a  visual  cue,  does  it 
match  cultural  conventions? 

OOO 

2.6 

Do  the  selected  colors  correspond  to 
common  expectations  about  color  codes? 

OOO 

2.7 

When  prompts  imply  a  necessary  action, 
are  the  words  in  the  message  consistent 
with  that  action? 

OOO 

2.8 

Do  keystroke  references  in  prompts  match 
actual  key  names? 

OOO 

2.9 

On  data  entry  screens,  are  tasks  described 
in  terminology  familiar  to  users? 

OOO 

2.1 

Are  field-level  prompts  provided  for  data 
entry  screens? 

OOO 

2.11 

For  question  and  answer  interfaces,  are 
questions  stated  in  clear,  simple 
language? 

OOO 

2.12 

Do  menu  choices  fit  logically  into 
categories  that  have  readily  understood 
meanings? 

OOO 

2.13 

Are  menu  titles  parallel  grammatically? 

OOO 

2.14 

Does  the  command  language  employ  user 
jargon  and  avoid  computer  jargon? 

OOO 

2.15 

Are  command  names  specific  rather  than 
general? 

OOO 
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2.16 

Does  the  command  language  allow  both 
full  names  and  abbreviations? 

OOO 

2.17 

Are  input  data  codes  meaningful? 

OOO 

2.18 

Have  uncommon  letter  sequences  been 
avoided  whenever  possible? 

OOO 

2.19 

Does  the  system  automatically  enter 
leading  or  trailing  spaces  to  align  decimal 
points? 

OOO 

2.2 

Does  the  system  automatically  enter  a 
dollar  sign  and  decimal  for  monetary 
entries? 

OOO 

2.21 

Does  the  system  automatically  enter 
commas  in  numeric  values  greater  than 
9999? 

OOO 

2.22 

Do  GUI  menus  offer  activation:  that  is, 
make  obvious  how  to  say  "now  do  it"? 

OOO 

2.23 

Has  the  system  been  designed  so  that  keys 
with  similar  names  do  not  perform 
opposite  (and  potentially  dangerous) 
actions? 

OOO 

2.24 

Are  function  keys  labeled  clearly  and 
distinctively,  even  if  this  means  breaking 
consistency  rules? 

OOO 

3.  User  Control  and  Freedom 

Users  should  be  free  to  select  and  sequence  tasks  (when  appropriate),  rather  than  having  the  system  do  this 
for  them.  Users  often  choose  system  functions  by  mistake  and  will  need  a  clearly  marked  "emergency  exit" 
to  leave  the  unwanted  state  without  having  to  go  through  an  extended  dialogue.  Users  should  make  their 
own  decisions  (with  clear  information)  regarding  the  costs  of  exiting  current  work.  The  system  should 
support  undo  and  redo. 


# 

Review  Checklist 

Y  NNA 

Comments 

3.1 

If  setting  up  windows  is  a  low-frequency 
task,  is  it  particularly  easy  to  remember? 

OOO 
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3.2 

In  systems  that  use  overlapping  windows, 
is  it  easy  for  users  to  rearrange  windows 
on  the  screen? 

OOO 

3.3 

In  systems  that  use  overlapping  windows, 
is  it  easy  for  users  to  switch  between 
windows? 

OOO 

3.4 

When  a  user's  task  is  complete,  does  the 
system  wait  for  a  signal  from  the  user 
before  processing? 

OOO 

3.5 

Can  users  type-ahead  in  a  system  with 
many  nested  menus? 

OOO 

3.6 

Are  users  prompted  to  confirm  commands 
that  have  drastic,  destructive 
consequences? 

OOO 

3.7 

Is  there  an  "undo"  function  at  the  level  of 
a  single  action,  a  data  entry,  and  a 
complete  group  of  actions? 

OOO 

3.8 

Can  users  cancel  out  of  operations  in 
progress? 

OOO 

3.9 

Are  character  edits  allowed  in 
commands? 

OOO 

3.1 

Can  users  reduce  data  entry  time  by 
copying  and  modifying  existing  data? 

OOO 

3.11 

Are  character  edits  allowed  in  data  entry 
fields? 

OOO 

3.12 

If  menu  lists  are  long  (more  than  seven 
items),  can  users  select  an  item  either  by 
moving  the  cursor  or  by  typing  a 
mnemonic  code? 

OOO 
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3.13 

If  the  system  uses  a  pointing  device,  do 
users  have  the  option  of  either  clicking  on 
menu  items  or  using  a  keyboard  shortcut? 

OOO 

3.14 

Are  menus  broad  (many  items  on  a  menu) 
rather  than  deep  (many  menu  levels)? 

OOO 

3.15 

If  the  system  has  multiple  menu  levels,  is 
there  a  mechanism  that  allows  users  to  go 
back  to  previous  menus? 

OOO 

3.16 

If  users  can  go  back  to  a  previous  menu, 
can  they  change  their  earlier  menu 
choice? 

OOO 

3.17 

Can  users  move  forward  and  backward 
between  fields  or  dialog  box  options? 

OOO 

3.18 

If  the  system  has  multipage  data  entry 
screens,  can  users  move  backward  and 
forward  among  all  the  pages  in  the  set? 

OOO 

3.19 

If  the  system  uses  a  question  and  answer 
interface,  can  users  go  back  to  previous 
questions  or  skip  forward  to  later 
questions? 

OOO 

3.2 

Do  function  keys  that  can  cause  serious 
consequences  have  an  undo  feature? 

OOO 

3.21 

Can  users  easily  reverse  their  actions? 

OOO 

3.22 

If  the  system  allows  users  to  reverse  their 
actions,  is  there  a  retracing  mechanism  to 
allow  for  multiple  undos? 

OOO 

3.23 

Can  users  set  their  own  system,  session, 
file,  and  screen  defaults? 

OOO 
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4.  Consistency  and  Standards 


Users  should  not  have  to  wonder  whether  different  words,  situations,  or  actions  mean  the  same  thing. 
Follow  platform  conventions. 


# 

Review  Checklist 

Y  NNA 

Comments 

4.1 

Have  industry  or  company  formatting 
standards  been  followed  consistently  in 
all  screens  within  a  system? 

ooo 

4.2 

Has  a  heavy  use  of  all  uppercase  letters  on 
a  screen  been  avoided? 

ooo 

4.3 

Do  abbreviations  not  include  punctuation? 

ooo 

4.4 

Are  integers  right -justified  and  real 
numbers  decimal-aligned? 

ooo 

4.5 

Are  icons  labeled? 

ooo 

4.6 

Are  there  no  more  than  twelve  to  twenty 
icon  types? 

ooo 

4.7 

Are  there  salient  visual  cues  to  identify 
the  active  window? 

ooo 

4.8 

Does  each  window  have  a  title? 

ooo 

4.9 

Are  vertical  and  horizontal  scrolling 
possible  in  each  window? 

ooo 

4.1 

Does  the  menu  structure  match  the  task 
structure? 

ooo 
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4.11 

Have  industry  or  company  standards  been 
established  for  menu  design,  and  are  they 
applied  consistently  on  all  menu  screens 
in  the  system? 

OOO 

4.12 

Are  menu  choice  lists  presented 
vertically? 

OOO 

4.13 

If  "exit"  is  a  menu  choice,  does  it  always 
appear  at  the  bottom  of  the  list? 

OOO 

4.14 

Are  menu  titles  either  centered  or  left- 
justified? 

OOO 

4.15 

Are  menu  items  left-justified,  with  the 
item  number  or  mnemonic  preceding  the 
name? 

OOO 

4.16 

Do  embedded  field-level  prompts  appear 
to  the  right  of  the  field  label? 

OOO 

4.17 

Do  on-line  instructions  appear  in  a 
consistent  location  across  screens? 

OOO 

4.18 

Are  field  labels  and  fields  distinguished 
typographically? 

OOO 

4.19 

Are  field  labels  consistent  from  one  data 
entry  screen  to  another? 

OOO 

4.2 

Are  fields  and  labels  left-justified  for 
alpha  lists  and  right-justified  for  numeric 
lists? 

OOO 

4.21 

Do  field  labels  appear  to  the  left  of  single 
fields  and  above  list  fields? 

OOO 
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4.22 

Are  attention-getting  techniques  used  with 
care? 

OOO 

4.23 

Intensity:  two  levels  only 

OOO 

4.24 

Size:  up  to  four  sizes 

OOO 

4.25 

Font:  up  to  three 

OOO 

4.26 

Blink:  two  to  four  hertz 

OOO 

4.27 

Color:  up  to  four  (additional  colors  for 
occasional  use  only) 

OOO 

4.28 

Sound:  soft  tones  for  regular  positive 
feedback,  harsh  for  rare  critical  conditions 

OOO 

4.29 

Are  attention-getting  techniques  used  only 
for  exceptional  conditions  or  for  time- 
dependent  information? 

OOO 

4.3 

Are  there  no  more  than  four  to  seven 
colors,  and  are  they  far  apart  along  the 
visible  spectrum? 

OOO 

4.31 

Is  a  legend  provided  if  color  codes  are 
numerous  or  not  obvious  in  meaning? 

OOO 

4.32 

Have  pairings  of  high-chroma,  spectrally 
extreme  colors  been  avoided? 

OOO 
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4.33 

Are  saturated  blues  avoided  for  text  or 
other  small,  thin  line  symbols? 

OOO 

4.34 

Is  the  most  important  information  placed 
at  the  beginning  of  the  prompt? 

OOO 

4.35 

Are  user  actions  named  consistently 
across  all  prompts  in  the  system? 

OOO 

4.36 

Are  system  objects  named  consistently 
across  all  prompts  in  the  system? 

OOO 

4.37 

Do  field-level  prompts  provide  more 
information  than  a  restatement  of  the  field 
name? 

OOO 

4.38 

For  question  and  answer  interfaces,  are 
the  valid  inputs  for  a  question  listed? 

OOO 

4.39 

Are  menu  choice  names  consistent,  both 
within  each  menu  and  across  the  system, 
in  grammatical  style  and  terminology? 

OOO 

4.4 

Does  the  structure  of  menu  choice  names 
match  their  corresponding  menu  titles? 

OOO 

4.41 

Are  commands  used  the  same  way,  and 
do  they  mean  the  same  thing,  in  all  parts 
of  the  system? 

OOO 

4.42 

Does  the  command  language  have  a 
consistent,  natural,  and  mnemonic  syntax? 

OOO 

4.43 

Do  abbreviations  follow  a  simple  primary 
rule  and,  if  necessary,  a  simple  secondary 
rule  for  abbreviations  that  otherwise 
would  be  duplicates? 

OOO 
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4.44 

Is  the  secondary  rule  used  only  when 
necessary? 

OOO 

4.45 

Are  abbreviated  words  all  the  same 
length? 

OOO 

4.46 

Is  the  structure  of  a  data  entry  value 
consistent  from  screen  to  screen? 

OOO 

4.47 

Is  the  method  for  moving  the  cursor  to  the 
next  or  previous  field  consistent 
throughout  the  system? 

OOO 

4.48 

If  the  system  has  multipage  data  entry 
screens,  do  all  pages  have  the  same  title? 

OOO 

4.49 

If  the  system  has  multipage  data  entry 
screens,  does  each  page  have  a  sequential 
page  number? 

OOO 

4.5 

Does  the  system  follow  industry  or 
company  standards  for  function  key 
assignments? 

OOO 

4.51 

Are  high-value,  high-chroma  colors  used 
to  attract  attention? 

OOO 

5.  Help  Users  Recognize,  Diagnose,  and  Recover  From  Errors 

Error  messages  should  be  expressed  in  plain  language  (no  codes). 


# 

Review  Checklist 

Y  NNA 

Comments 

5.1 

Is  sound  used  to  signal  an  error? 

OOO 

5.2 

Are  prompts  stated  constructively, 
without  overt  or  implied  criticism  of  the 
user? 

OOO 
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5.3 

Do  prompts  imply  that  the  user  is  in 
control? 

OOO 

5.4 

Are  prompts  brief  and  unambiguous. 

OOO 

5.5 

Are  error  messages  worded  so  that  the 
system,  not  the  user,  takes  the  blame? 

OOO 

5.6 

If  humorous  error  messages  are  used,  are 
they  appropriate  and  inoffensive  to  the 
user  population? 

OOO 

5.7 

Are  error  messages  grammatically 
correct? 

OOO 

5.8 

Do  error  messages  avoid  the  use  of 
exclamation  points? 

OOO 

5.9 

Do  error  messages  avoid  the  use  of 
violent  or  hostile  words? 

OOO 

5.1 

Do  error  messages  avoid  an 
anthropomorphic  tone? 

OOO 

5.11 

Do  all  error  messages  in  the  system  use 
consistent  grammatical  style,  form, 
terminology,  and  abbreviations? 

OOO 

5.12 

Do  messages  place  users  in  control  of  the 
system? 

OOO 

5.13 

Does  the  command  language  use  normal 
action-object  syntax? 

OOO 

5.14 

Does  the  command  language  avoid 
arbitrary,  non-English  use  of  punctuation, 
except  for  symbols  that  users  already 
know? 

OOO 
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5.15 

If  an  error  is  detected  in  a  data  entry  field, 
does  the  system  place  the  cursor  in  that 
field  or  highlight  the  error? 

OOO 

5.16 

Do  error  messages  inform  the  user  of  the 
error's  severity? 

OOO 

5.17 

Do  error  messages  suggest  the  cause  of 
the  problem? 

OOO 

5.18 

Do  error  messages  provide  appropriate 
semantic  information? 

OOO 

5.19 

Do  error  messages  provide  appropriate 
syntactic  information? 

OOO 

5.2 

Do  error  messages  indicate  what  action 
the  user  needs  to  take  to  correct  the  error? 

OOO 

5.21 

If  the  system  supports  both  novice  and 
expert  users,  are  multiple  levels  of  error- 
message  detail  available? 

OOO 

6.  Error  Prevention 


Even  better  than  good  error  messages  is  a  careful  design  which  prevents  a  problem  from  occurring  in  the 
first  place. 


# 

Review  Checklist 

Y  NNA 

Comments 

6.1 

If  the  database  includes  groups  of  data, 
can  users  enter  more  than  one  group  on  a 
single  screen? 

OOO 

6.2 

Have  dots  or  underscores  been  used  to 
indicate  field  length? 

OOO 

6.3 

Is  the  menu  choice  name  on  a  higher-level 
menu  used  as  the  menu  title  of  the  lower- 
level  menu? 

OOO 
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6.4 

Are  menu  choices  logical,  distinctive,  and 
mutually  exclusive? 

OOO 

6.5 

Are  data  inputs  case-blind  whenever 
possible? 

OOO 

6.6 

If  the  system  displays  multiple  windows, 
is  navigation  between  windows  simple 
and  visible? 

OOO 

6.7 

Are  the  function  keys  that  can  cause  the 
most  serious  consequences  in  hard-to- 
reach  positions? 

OOO 

6.8 

Are  the  function  keys  that  can  cause  the 
most  serious  consequences  located  far 
away  from  low-consequence  and  high-use 
keys? 

OOO 

6.9 

Has  the  use  of  qualifier  keys  been 
minimized? 

OOO 

6.1 

If  the  system  uses  qualifier  keys,  are  they 
used  consistently  throughout  the  system? 

OOO 

6.11 

Does  the  system  prevent  users  from 
making  errors  whenever  possible? 

OOO 

6.12 

Does  the  system  warn  users  if  they  are 
about  to  make  a  potentially  serious  error? 

OOO 

6.13 

Does  the  system  intelligently  interpret 
variations  in  user  commands? 

OOO 

6.14 

Do  data  entry  screens  and  dialog  boxes 
indicate  the  number  of  character  spaces 
available  in  a  field? 

OOO 

6.15 

Do  fields  in  data  entry  screens  and  dialog 
boxes  contain  default  values  when 
appropriate? 

OOO 
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7.  Recognition  Rather  Than  Recall 


Make  objects,  actions,  and  options  visible.  The  user  should  not  have  to  remember  information  from  one 
part  of  the  dialogue  to  another.  Instructions  for  use  of  the  system  should  be  visible  or  easily  retrievable 
whenever  appropriate. 


# 

Review  Checklist 

Y  NNA 

Comments 

7.1 

For  question  and  answer  interfaces,  are 
visual  cues  and  white  space  used  to 
distinguish  questions,  prompts, 
instructions,  and  user  input? 

ooo 

7.2 

Does  the  data  display  start  in  the  upper- 
left  corner  of  the  screen? 

ooo 

7.3 

Are  multiword  field  labels  placed 
horizontally  (not  stacked  vertically)? 

ooo 

7.4 

Are  all  data  a  user  needs  on  display  at 
each  step  in  a  transaction  sequence? 

ooo 

7.5 

Are  prompts,  cues,  and  messages  placed 
where  the  eye  is  likely  to  be  looking  on 
the  screen? 

ooo 

7.6 

Have  prompts  been  formatted  using  white 
space,  justification,  and  visual  cues  for 
easy  scanning? 

ooo 

7.7 

Do  text  areas  have  "breathing  space" 
around  them? 

ooo 

7.8 

Is  there  an  obvious  visual  distinction 
made  between  "choose  one"  menu  and 
"choose  many"  menus? 

ooo 

7.9 

Have  spatial  relationships  between  soft 
function  keys  (on-screen  cues)  and 
keyboard  function  keys  been  preserved? 

ooo 

7.1 

Does  the  system  gray  out  or  delete  labels 
of  currently  inactive  soft  function  keys? 

ooo 
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7.11 

Is  white  space  used  to  create  symmetry 
and  lead  the  eye  in  the  appropriate 
direction? 

OOO 

7.12 

Have  items  been  grouped  into  logical 
zones,  and  have  headings  been  used  to 
distinguish  between  zones? 

OOO 

7.13 

Are  zones  no  more  than  twelve  to 
fourteen  characters  wide  and  six  to  seven 
lines  high? 

OOO 

7.14 

Have  zones  been  separated  by  spaces, 
lines,  color,  letters,  bold  titles,  rules  lines, 
or  shaded  areas? 

OOO 

7.15 

Are  field  labels  close  to  fields,  but 
separated  by  at  least  one  space? 

OOO 

7.16 

Are  long  columnar  fields  broken  up  into 
groups  of  five,  separated  by  a  blank  line? 

OOO 

7.17 

Are  optional  data  entry  fields  clearly 
marked? 

OOO 

7.18 

Are  symbols  used  to  break  long  input 
strings  into  "chunks"? 

OOO 

7.19 

Is  reverse  video  or  color  highlighting  used 
to  get  the  user's  attention? 

OOO 

7.2 

Is  reverse  video  used  to  indicate  that  an 
item  has  been  selected? 

OOO 

7.21 

Are  size,  boldface,  underlining,  color, 
shading,  or  typography  used  to  show 
relative  quantity  or  importance  of 
different  screen  items? 

OOO 

7.22 

Are  borders  used  to  identify  meaningful 
groups? 

OOO 
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7.23 

Has  the  same  color  been  used  to  group 
related  elements? 

OOO 

7.24 

Is  color  coding  consistent  throughout  the 
system? 

OOO 

7.25 

Is  color  used  in  conjunction  with  some 
other  redundant  cue? 

OOO 

7.26 

Is  there  good  color  and  brightness  contrast 
between  image  and  background  colors? 

OOO 

7.27 

Have  light,  bright,  saturated  colors  been 
used  to  emphasize  data  and  have  darker, 
duller,  and  desaturated  colors  been  used 
to  de-emphasize  data? 

OOO 

7.28 

Is  the  first  word  of  each  menu  choice  the 
most  important? 

OOO 

7.29 

Does  the  system  provide  mapping:  that  is, 
are  the  relationships  between  controls  and 
actions  apparent  to  the  user? 

OOO 

7.3 

Are  input  data  codes  distinctive? 

OOO 

7.31 

Have  frequently  confused  data  pairs  been 
eliminated  whenever  possible? 

OOO 

7.32 

Have  large  strings  of  numbers  or  letters 
been  broken  into  chunks? 

OOO 

7.33 

Are  inactive  menu  items  grayed  out  or 
omitted? 

OOO 

7.34 

Are  there  menu  selection  defaults? 

OOO 
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7.35 

If  the  system  has  many  menu  levels  or 
complex  menu  levels,  do  users  have 
access  to  an  on-line  spatial  menu  map? 

OOO 

7.36 

Do  GUI  menus  offer  affordance:  that  is, 
make  obvious  where  selection  is  possible? 

OOO 

7.37 

Are  there  salient  visual  cues  to  identify 
the  active  window? 

OOO 

7.38 

Are  function  keys  arranged  in  logical 
groups? 

OOO 

7.39 

Do  data  entry  screens  and  dialog  boxes 
indicate  when  fields  are  optional? 

OOO 

7.4 

On  data  entry  screens  and  dialog  boxes, 
are  dependent  fields  displayed  only  when 
necessary? 

OOO 

8.  Flexibility  and  Minimalist  Design 

Accelerators-unseen  by  the  novice  user-may  often  speed  up  the  interaction  for  the  expert  user  such  that  the 
system  can  cater  to  both  inexperienced  and  experienced  users.  Allow  users  to  tailor  frequent  actions. 
Provide  alternative  means  of  access  and  operation  for  users  who  differ  from  the  "average"  user  (e.g., 
physical  or  cognitive  ability,  culture,  language,  etc.) 


# 

Review  Checklist 

Y  NNA 

Comments 

8.1 

If  the  system  supports  both  novice  and 
expert  users,  are  multiple  levels  of  error 
message  detail  available? 

OOO 

8.2 

Does  the  system  allow  novices  to  use  a 
keyword  grammar  and  experts  to  use  a 
positional  grammar? 

OOO 

8.3 

Can  users  define  their  own  synonyms  for 
commands? 

OOO 

8.4 

Does  the  system  allow  novice  users  to 
enter  the  simplest,  most  common  form  of 
each  command,  and  allow  expert  users  to 
add  parameters? 

OOO 
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8.5 

Do  expert  users  have  the  option  of 
entering  multiple  commands  in  a  single 
string? 

OOO 

8.6 

Does  the  system  provide  function  keys  for 
high-frequency  commands? 

OOO 

8.7 

For  data  entry  screens  with  many  fields  or 
in  which  source  documents  may  be 
incomplete,  can  users  save  a  partially 
filled  screen? 

OOO 

8.8 

Does  the  system  automatically  enter 
leading  zeros? 

OOO 

8.9 

If  menu  lists  are  short  (seven  items  or 
fewer),  can  users  select  an  item  by 
moving  the  cursor? 

OOO 

8.1 

If  the  system  uses  a  type-ahead  strategy, 
do  the  menu  items  have  mnemonic  codes? 

OOO 

8.11 

If  the  system  uses  a  pointing  device,  do 
users  have  the  option  of  either  clicking  on 
fields  or  using  a  keyboard  shortcut? 

OOO 

8.12 

Does  the  system  offer  "find  next"  and 
"find  previous"  shortcuts  for  database 
searches? 

OOO 

8.13 

On  data  entry  screens,  do  users  have  the 
option  of  either  clicking  directly  on  a  field 
or  using  a  keyboard  shortcut? 

OOO 

8.14 

On  menus,  do  users  have  the  option  of 
either  clicking  directly  on  a  menu  item  or 
using  a  keyboard  shortcut? 

OOO 

8.15 

In  dialog  boxes,  do  users  have  the  option 
of  either  clicking  directly  on  a  dialog  box 
option  or  using  a  keyboard  shortcut? 

OOO 

8.16 

Can  expert  users  bypass  nested  dialog 
boxes  with  either  type-ahead,  user- 
defined  macros,  or  keyboard  shortcuts? 

OOO 
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9.  Aesthetic  and  Minimalist  Design 


Dialogues  should  not  contain  information  which  is  irrelevant  or  rarely  needed.  Every  extra  unit  of 
information  in  a  dialogue  competes  with  the  relevant  units  of  information  and  diminishes  their  relative 
visibility. 


# 

Review  Checklist 

Y  NNA 

Comments 

9.1 

Is  only  (and  all)  information  essential  to 
decision  making  displayed  on  the  screen? 

ooo 

9.2 

Are  all  icons  in  a  set  visually  and 
conceptually  distinct? 

ooo 

9.3 

Have  large  objects,  bold  lines,  and  simple 
areas  been  used  to  distinguish  icons? 

ooo 

9.4 

Does  each  icon  stand  out  from  its 
background? 

ooo 

9.5 

If  the  system  uses  a  standard  GUI 
interface  where  menu  sequence  has 
already  been  specified,  do  menus  adhere 
to  the  specification  whenever  possible? 

ooo 

9.6 

Are  meaningful  groups  of  items  separated 
by  white  space? 

ooo 

9.7 

Does  each  data  entry  screen  have  a  short, 
simple,  clear,  distinctive  title? 

ooo 

9.8 

Are  field  labels  brief,  familiar,  and 
descriptive? 

ooo 

9.9 

Are  prompts  expressed  in  the  affirmative, 
and  do  they  use  the  active  voice? 

ooo 

9.1 

Is  each  lower-level  menu  choice 
associated  with  only  one  higher  level 
menu? 

ooo 

9.11 

Are  menu  titles  brief,  yet  long  enough  to 
communicate? 

ooo 

106 


Are  there  pop-up  or  pull-down  menus 
within  data  entry  fields  that  have  many. 

9.12 

but  well-defined,  entry  options? 

OOO 

10.  Help  and  Documentation 


Even  though  it  is  better  if  the  system  can  be  used  without  documentation,  it  may  be  necessary  to  provide 
help  and  documentation.  Any  such  information  should  be  easy  to  search,  focused  on  the  user’s  task,  list 
concrete  steps  to  be  carried  out,  and  not  be  too  large. _ _ 


# 

Review  Checklist 

Y  NNA 

Comments 

10.1 

If  users  are  working  from  hard  copy,  are 
the  parts  of  the  hard  copy  that  go  on-line 
marked? 

OOO 

10.2 

Are  on-line  instructions  visually  distinct? 

OOO 

10.3 

Do  the  instructions  follow  the  sequence  of 
user  actions? 

OOO 

10.4 

If  menu  choices  are  ambiguous,  does  the 
system  provide  additional  explanatory 
information  when  an  item  is  selected? 

OOO 

10.5 

Are  data  entry  screens  and  dialog  boxes 
supported  by  navigation  and  completion 
instructions? 

OOO 

10.6 

If  menu  items  are  ambiguous,  does  the 
system  provide  additional  explanatory 
information  when  an  item  is  selected? 

OOO 

10.7 

Are  there  memory  aids  for  commands, 
either  through  on-line  quick  reference  or 
prompting? 

OOO 

10.8 

Is  the  help  function  visible;  for  example,  a 
key  labeled  HELP  or  a  special  menu? 

OOO 

10.9 

Is  the  help  system  interface  (navigation, 
presentation,  and  conversation)  consistent 
with  the  navigation,  presentation,  and 
conversation  interfaces  of  the  application 
it  supports? 

OOO 
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10.1 

Navigation:  Is  information  easy  to  find? 

OOO 

10.1 

Presentation:  Is  the  visual  layout  well 
designed? 

OOO 

10.1 

Conversation:  Is  the  information  accurate, 
complete,  and  understandable? 

OOO 

10.1 

Is  the  information  relevant? 

OOO 

10.1 

Goal-oriented  (What  can  I  do  with  this 
program?) 

OOO 

10.2 

Descriptive  (What  is  this  thing  for?) 

OOO 

10.2 

Procedural  (How  do  I  do  this  task?) 

OOO 

10.2 

Interpretive  (Why  did  that  happen?) 

OOO 

10.2 

Navigational  (Where  am  I?) 

OOO 

10.2 

Is  there  context-sensitive  help? 

OOO 

10.2 

Can  the  user  change  the  level  of  detail 
available? 

OOO 

10.2 

Can  users  easily  switch  between  help  and 
their  work? 

OOO 
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10.2 

Is  it  easy  to  access  and  return  from  the 
help  system? 

OOO 

10.2 

Can  users  resume  work  where  they  left 
off  after  accessing  help? 

OOO 

11.  Skills 

The  system  should  support,  extend,  supplement,  or  enhance  the  user’s  skills,  background  knowledge,  and 
expertise  — not  replace  them. 


# 

Review  Checklist 

Y  NNA 

Comments 

11.1 

Can  users  choose  between  iconic  and  text 
display  of  information? 

OOO 

11.2 

Are  window  operations  easy  to  learn  and 
use? 

OOO 

11.3 

If  users  are  experts,  usage  is  frequent,  or 
the  system  has  a  slow  response  time,  are 
there  fewer  screens  (more  information  per 
screen)? 

OOO 

11.4 

If  users  are  novices,  usage  is  infrequent, 
or  the  system  has  a  fast  response  time,  are 
there  more  screens  (less  information  per 
screen)? 

OOO 

11.5 

Does  the  system  automatically  color-code 
items,  with  little  or  no  user  effort? 

OOO 

11.6 

If  the  system  supports  both  novice  and 
expert  users,  are  multiple  levels  of  detail 
available. 

OOO 

11.7 

Are  users  the  initiators  of  actions  rather 
than  the  responders? 

OOO 

11.8 

Does  the  system  perform  data  translations 
for  users? 

OOO 
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11.9 

Do  field  values  avoid  mixing  alpha  and 
numeric  characters  whenever  possible? 

OOO 

11.1 

If  the  system  has  deep  (multilevel)  menus, 
do  users  have  the  option  of  typing  ahead? 

OOO 

11.1 

When  the  user  enters  a  screen  or  dialog 
box,  is  the  cursor  already  positioned  in  the 
field  users  are  most  likely  to  need? 

OOO 

11.1 

Can  users  move  forward  and  backward 
within  a  field? 

OOO 

11.1 

Is  the  method  for  moving  the  cursor  to  the 
next  or  previous  field  both  simple  and 
visible? 

OOO 

11.2 

Has  auto-tabbing  been  avoided  except 
when  fields  have  fixed  lengths  or  users 
are  experienced? 

OOO 

11.2 

Do  the  selected  input  device(s)  match  user 
capabilities? 

OOO 

11.2 

Are  cursor  keys  arranged  in  either  an 
inverted  T  (best  for  experts)  or  a  cross 
configuration  (best  for  novices)? 

OOO 

11.2 

Are  important  keys  (for  example,  ENTER 
,  TAB)  larger  than  other  keys? 

OOO 

11.2 

Are  there  enough  function  keys  to  support 
functionality,  but  not  so  many  that 
scanning  and  finding  are  difficult? 

OOO 

11.2 

Are  function  keys  reserved  for  generic, 
high-frequency,  important  functions? 

OOO 

11.2 

Are  function  key  assignments  consistent 
across  screens,  subsystems,  and  related 
products? 

OOO 
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Does  the  system  correctly  anticipate  and 
prompt  for  the  user's  probable  next 

11.2 

activity? 

OOO 

12.  Pleasurable  and  Respectful  Interaction  with  the  User 


The  user’s  interactions  with  the  system  should  enhance  the  quality  of  her  or  his  work-life.  The  user  should 
be  treated  with  respect.  The  design  should  be  aesthetically  pleasing-  with  artistic  as  well  as  functional 
value. 


# 

Review  Checklist 

Y  NNA 

Comments 

12.1 

Is  each  individual  icon  a  harmonious 
member  of  a  family  of  icons? 

OOO 

12.2 

Has  excessive  detail  in  icon  design  been 
avoided? 

OOO 

12.3 

Has  color  been  used  with  discretion? 

OOO 

12.4 

Has  the  amount  of  required  window 
housekeeping  been  kept  to  a  minimum? 

OOO 

12.5 

If  users  are  working  from  hard  copy,  does 
the  screen  layout  match  the  paper  form? 

OOO 

12.6 

Has  color  been  used  specifically  to  draw 
attention,  communicate  organization, 
indicate  status  changes,  and  establish 
relationships? 

OOO 

12.7 

Can  users  turn  off  automatic  color  coding 
if  necessary? 

OOO 

12.8 

Are  typing  requirements  minimal  for 
question  and  answer  interfaces? 

OOO 

12.9 

Do  the  selected  input  device(s)  match 
environmental  constraints? 

OOO 
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12.1 

If  the  system  uses  multiple  input  devices, 
has  hand  and  eye  movement  between 
input  devices  been  minimized? 

OOO 

12.1 

If  the  system  supports  graphical  tasks,  has 
an  alternative  pointing  device  been 
provided? 

OOO 

12.2 

Is  the  numeric  keypad  located  to  the  right 
of  the  alpha  key  area? 

OOO 

12.2 

Are  the  most  frequently  used  function 
keys  in  the  most  accessible  positions? 

OOO 

12.2 

Does  the  system  complete  unambiguous 
partial  input  on  a  data  entry  field? 

OOO 

Note  to  Case  Study  Observer 

Using  the  space  provided  below,  please  provide  information  about  your 
background  and  previous  experience  regarding  information  technology,  information 
systems,  and  heuristic  inspection  methods. 
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Data  Reporting  Procedures 

The  case  study  observer  will  submit  the  data  collected  to  the  researcher  in 
whichever  form  is  most  convenient  for  the  observer.  This  submittal  should  include  the 
case  study  instrument  (the  heuristic  checklist)  as  well  as  the  observer’s  background 
information  related  to  information  technology  and  information  systems. 


113 


Bibliography 


lcog.com.  "Don't  Say  'Search  Engine'  -  Say  'Google'",  n.pag. 

http://www.lcog.com/search-engine-statistics.html.  November  2004. 

Adams,  Dennis  A.,  Ryan  R.  Nelson,  and  Peter  A.  Todd.  “Perceived  Usefulness,  Ease  of 
Use,  and  Usage  of  Information  Technology:  A  Replication.”  MIS  Quarterly,  16, 

2:  227-247  (June  1992). 

Agarwal,  Ritu  and  Jayesh  Prasad.  “Are  Individual  Differences  Germane  to  the 

Acceptance  of  New  Information  Technologies?”  Decision  Sciences,  30,  2:  361- 
391  (Spring  1999). 

Air  Force  Civil  Engineer  Support  Agency  (AFCESA).  Automated  Civil  Engineer  System 
(ACES)  Command,  Control,  Communications,  Computers,  and  Intelligence 
Support  Plan  (C4ISP).  Tyndall  AFB:  HQ  AFCESA,  Sep  2002. 

Air  Force  Civil  Engineer  Support  Agency  (AFCESA).  Automated  Civil  Engineer  System 
System  Engineering  Process  Guide.  Tyndall  AFB:  HQ  AFCESA,  Aug  2001. 

Air  Force  Civil  Engineer  Support  Agency  (AFCESA).  Concept  of  Operations 

(CONOPS)  for  the  Automated  Civil  Engineer  System  (ACES).  Tyndall  AFB:  HQ 
AFCESA,  1  May  2003. 

Air  Force  Reserve  Command  (AFRC).  C4  Systems  Requirements  Document  CSO 
Control  Number  AFRC2003-006.  Robins  AFB:  HQ  AFRC,  1  Jul  2003. 

Davis,  Fred  D.,  Bagozzi,  Richard  P.,  and  Warshaw,  Paul  R.,  “User  Acceptance  of 

Computer  Technology:  A  Comparison  of  Two  Theoretical  Models”.  The  Institute 
of  Management  Sciences.  1989,  pages  982-1003. 

Davis,  Fred  D.  and  Viswanath  Venkatesh.  “Toward  Preprototype  User  Acceptance 
Testing  of  New  Information  Systems:  Implications  for  Software  Project 
Management.”  IEEE  Transactions  on  Engineering  Management,  51,1:  31-46 
(February  2004). 

Department  of  Defense  (DoD).  Department  of  Defense  (DoD)  Joint  Technical 
Architecture  (JTA)  Version  6.0.  Washington,  D.C.  24  November  2003. 

Department  of  Defense  (DoD).  Interoperability  and  Supportability  of  Information 

Technology  (IT)  and  National  Security  Systems  (NSS).  DoD  Directive  4630.5 
Washington,  GPO,  5  May  2004. 


114 


Department  of  Defense  (DoD).  Technical  Architecture  Framework  for  Information 

Management,  Volume  8:  Human  Computer  Interface  Style  Guide.  Washington: 
GPO,  30  Apr  1996. 

Department  of  the  Air  Force  (HQ  USAF).  Full  Spectrum  Threat  Response  (FSTR) 

Planning  and  Operations.  AFI  10-2501.  Washington:  HQ  USAF,  24  Dec  2002. 

Department  of  the  Air  Force  (HQ  USAF).  Prime  Base  Engineer  Emergency  Force 
(BEEF)  Program.  AFI  10-210.  Washington:  HQ  USAF,  1  Oct  2004. 

Disronline.disa.mil.  "DISR  Reports  and  Archives."  n.pag. 
http :  //di  sronline .  dis  a .  mil/ V  JT  A/j  t  a_report  s .  j  sp 

Fourteen  Heuristics  [OCFC  Heuristic  Evaluation]  www.oclc.org.  ©2004  OCFC  Online 

Computer  Fibrary  Center,  http://www.oclc.org/policies/usability/heuristic/set.htm. 
30  May  2004. 

Garcia,  Carolyne.  “Simple.  Obvious.  Intuitive.  Impossible.”  n.pag. 

http://advancement.uark.edu/pubs/Research_Frontiers/fall_2003/08F3.html. 
February  2005. 

Graziano,  Anthony  M.  and  Michael  F.  Raulin.  Research  Methods:  A  Process  of  Inquiry 
(5th  Edition).  Boston:  Pearson,  2004. 

Hendrickson,  Anthony  R.,  Patti  D.  Massey,  and  Timothy  Paul  Cronan.  “On  the  Test- 
Retest  Reliability  of  Perceived  Usefulness  and  Perceived  Ease  of  Use  Scales.” 

MIS  Quarterly,  17,  2:  227-230  (June  1993). 

Hoffer,  Jeffrey  A.,  Joey  F.  George,  and  Joseph  S.  Valacich.  Modern  Systems  Analysis 
and  Design  (3rd  Edition).  Prentice  Hall,  2001. 

Headquarters  System  Support  Group  (HQ  SSG)  (a).  “Automated  Civil  Engineer  System 
Personnel  Readiness  Module  Software  Test  Plan  (Version  2.0).”  Gunter  Annex, 
HQ  SSG:  2001. 

Headquarters  System  Support  Group  (HQ  SSG)  (b).  “Automated  Civil  Engineer  System 
Personnel  Readiness  Module  System  Design  Document  (Version  2.0).”  Gunter 
Annex,  HQ  SSG:  22  Feb  2001 . 

International  Organization  for  Standardization.  Ergonomic  requirements  for  office  work 
with  visucd  display  terminals  (VDTs)  --  Part  11:  Guidance  on  usability.  ISO  9241 
Part  11.  1998. 


115 


Malhotra,  Yogesh  and  Dennis  F.  Galletta.  “Extending  the  Technology  Acceptance 
Model  to  Account  for  Social  Influence:  Theoretical  Bases  and  Empirical 
Validation.”  In  Proceedings  of  the  32nd  Hawaii  International  Conference  on 
System  Sciences.  IEEE,  1999,  pages  1-14. 

McFarland,  Daniel  J.  and  Diane  Hamilton.  “Adding  Contextual  Specificity  to  the 
Technology  Acceptance  Model.”  Article  in  press,  Computers  in  Human 
Behavior:  1-21  (2004). 

Merriam-Webster,  Incorporated.  Merriam-Webster’s  Collegiate  Dictionary  (10th 
Edition ).  Springfield  MA:  Merriam-Webster,  Incorporated,  1994. 

Nielsen,  Jakob.  Designing  Web  Usability.  Indianapolis:  New  Riders  Publishing,  2000. 

Nielsen,  Jakob.  "Heuristic  Evaluation",  in  Usability  Inspection  Methods.  Ed.  by  J. 
Nielsen  and  R.  Mack.  John  Wiley  and  Sons,  Inc.  1994,  pages  25-62. 

Nielsen,  Jakob  and  Molich,  Rolf.  “Heuristic  Evaluation  of  User  Interfaces”,  in 

Proceedings  of  ACM  CHI’ 90  Conference  of  Human  Factors  in  Computing 
Systems.  ACM,  New  York.  1990,  pages  249-256. 

Nielsen,  Jakob  and  Tahir,  Marie.  “Homepage  Usability  50  Websites  Deconstructed”. 
New  Riders  Publishing  2002. 

Pierotti,  Deniese.  “Heuristic  Evaluation  -  A  System  Checklist”,  n.  pag. 

http://www.stcsig.org/usability/topics/articles/he-checklist.html.  15  December 
2002. 

Searchengineguide.com.  "Search  Engine  Guide."  n.  pag. 

http://www.searchengineguide.com.  November  2004. 

Szajna,  Bernadette.  “Software  Evaluation  and  Choice:  Predictive  Validation  of  the 

Technology  Acceptance  Instrument.”  MIS  Quarterly,  18,  3:  319-324  (September 
1994). 

Teamwarfare.com.  "Teamwarfare:  Community  Based  Gaming."  n.  pag. 
http://www.teamwarfare.com.  November  2004. 

Useit.com.  "About  Jakob  Nielsen."  n.  pag.  http://www.useit.com/jakob/index.html. 
November  2004. 

Useit.com.  “Guerrilla  HCI:  Using  Discount  Usability  Engineering  to  Penetrate  the 

Intimidation  Barrier”  n.  pag.  http://www.useit.com/papers/guerrilla_hci.html. 
1994. 


116 


Useit.com.  "Usability  101:  Introduction  to  Usability."  n.  pag. 

http://www.useit.com/alertbox/20030825.html.  25  August  2003. 

Venkatesh,  Viswanath  and  Fred  D.  Davis.  “A  Theoretical  Extension  of  the  Technology 
Acceptance  Model:  Four  Longitudinal  Field  Studies,”  Management  Science,  46, 

2:  186-204  (February  2000). 

Venkatesh,  Viswanath,  Cheri  Speier,  and  Michael  G.  Morris.  “User  Acceptance  Enablers 
in  Individual  Decision  Making  About  Technology:  Toward  an  Integrated  Model,” 
Decision  Sciences,  33,  2:  297-316  (Spring  2002). 

Venkatesh,  Viswanath,  Michael  G.  Morris,  Gordon  B.  Davis,  and  Fred  D.  Davis.  “User 
Acceptance  of  Information  Technology:  Toward  a  Unified  View,”  27,  3:  425-478 
(September  2003). 

Yin,  Robert  K.  Case  Study  Research  :  Design  and  Methods  (3rd  Edition).  Newbury  Park: 
Sage  Publications,  2002. 


117 


Vita 


Captain  Gunther  Kastenholz  graduated  from  Capital  High  School  in  Olympia, 
Washington.  He  entered  undergraduate  studies  at  the  University  of  Portland  in  Portland, 
Oregon  where  he  graduated  with  a  Bachelor  of  Science  degree  in  Mechanical 
Engineering  in  May  2000.  He  was  commissioned  as  a  Second  Lieutenant  upon 
completion  of  the  Detachment  695,  University  of  Portland,  Air  Force  Reserve  Officer 
Training  Corps  program. 

His  first  assignment  was  at  Minot  Air  Force  Base  in  July  2000.  At  this 
assignment  he  served  with  the  5th  Civil  Engineer  Squadron  as  a  mechanical  design 
engineer,  Readiness  Flight  officer,  and  personnel  section  commander.  In  August  2003, 
he  entered  the  Graduate  School  of  Engineering  and  Management,  Air  Force  Institute  of 
Technology.  Upon  graduation,  he  will  be  assigned  to  Los  Angeles  Air  Force  Base, 
California. 


118 


Form  Approved 

_ REPORT  DOCUMENTATION  PAGE _  I  OMB  No.  074-0188 

The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  the  collection  of  information,  including 
suggestions  for  reducing  this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite 
1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  an  penalty  for  failing  to  comply  with  a  collection  of  information  if 
it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1.  REPORT  DATE  (DD-MM-YYYY)  2.  REPORT  TYPE  3.  DATES  COVERED  (From  -  To) 

21-03-2005  Master’s  Thesis  Jun  2004  -  March  2005 


4.  TITLE  AND  SUBTITLE 

5a.  C 

CONTRACT  NUMBER 

5b.  C 

3RANT  NUMBER 

5c.  F 

•ROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  F 

>ROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAMES(S)  AND  ADDRESS(S) 

Air  Force  Institute  of  Technology 

Graduate  School  of  Engineering  and  Management  (AFIT/EN) 

2950  Hobson  Way 

WPAFB  OH  45433-7765 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

AFIT/GEM/ENV/05M-07 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

N/A 

10.  SPONSOR/MONITOR’S 

ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

This  research  used  a  case  study  methodology  where  one  instrument  used  to  collect  data  was  a  heuristic  usability  inspection 
method.  Data  was  collected  from  a  literature  review  of  pertinent  Civil  Engineer  information  system  design  documents,  and 
conclusions  drawn  about  the  existing  level  of  specification  of  usability  engineering  principles.  The  heuristic  usability 
inspection  method  was  used  to  validate  the  conclusions  drawn  through  inspection  of  an  information  system  representative 
of  other  Civil  Engineer  information  systems,  the  Automated  Civil  Engineer  System  Personnel  Readiness  module. 


15.  SUBJECT  TERMS 

Usability  Engineering,  Technology  Acceptance,  Automated  Civil  Engineer  System  Personnel  Readiness  Module  (ACES  PR),  Heuristic  Inspection 


16.  SECURITY  CLASSIFICATION  OF: 
Unclassified 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 

OF 

PAGES 

19a.  NAME  OF  RESPONSIBLE  PERSON 

Alfred  E.  Thai,  Jr.,  (ENV) 

REPORT 

u 

ABSTRACT 

u 

c.  THIS  PAGE 

u 

uu 

130 

19b.  TELEPHONE  NUMBER  (Include  area  code) 

(937)  255-3636.  ext.  4798;  e-mail:  Alfred.Thal@afit.edu 

Standard  Form  298 (Rev:  8-98) 

Prescribed  by  ANSI  Sid  Z39-1 8 


119 


